Accessing HTTP header parameters to rest services in oracle bpel

One of the common requirement with rest services is to access/set the variable values in http header. In oracle SOA (BPEL) follow the below steps to access (for inbound rest) or set (for outbound) the HTTP header values,

For inbound rest service,

• Open the Receive Input activity, click on properties tab and add property with below syntax and assign it to a variable in your bpel process.

rest.binding.http.<variable_name>

For outbound invocations,

• Open the invoke rest activity, click on properties tab and add property with the same syntax as above and set the value of it from variable or expression.

Choosing Right Database For Your Application

Any applications performance largely depends on the database. Earlier there is nothing like choosing the database as mostly available option is relational database and you have to implement your requirements with the available option of RDBMS which leads to unexpected issues as you try to use RDBMS for a different type of data requirement. 

Now we have several databases options available which are build to serve the specific purpose/requirement and they are often called purpose build databases. Choosing the right database for you application is extremely critical for the success of your application. In this post I will try to throw some light on the different purpose built databases along with their use cases and offerings from different vendors. So lets get started,

Relational Model (RDBMS):
Traditional relational database systems which has the characteristics like,

• Data is stored in tables and rows and is normalised.
• Strict Schema and relationships via keys.
• Data accuracy, consistency and integrity.
• supports complex queries.

Examples : Oracle , mysql, Amazon aurora (cloud native)

Key value data stores:

Data will be stored as simple key value pairs and portioned by keys. Resilient to failure with high throughput and low latency reads and rights. Also offers consistent performance at scale.

Examples: Oracle NoSQL database, Amazon dynamoDB

Document Databases:

Data is stored in json like documents. Documents map naturally to how humans model data with flexible schema and indexing. Evolution of document databases,

• JSON became the de facto standard for data interchange.
• Initially json is being converted to RDBMS format and stored.
• There is a friction during this conversion.
• Object relational mappings (ORM's) were created to help with this friction.
• And then document databases solved the problem.

Examples: mongodb, couchdb, amazon documentdb.

In Memory data stores:

Data will be  stored in memory, it will not be persisted. Provides micros second performance. Provides simple commands to manipulate in memory data structures.

Examples: Redis, memcached, Amazon elastic cache

Search Databases:

Provides full text search engine functionality. Typical use cases are log analytics, full text search engine and stream analytics.

Examples: Aurweb, Amazon elastic search engine

Graph Databases:

Graph databases portray the data as it is viewed conceptually. This is accomplished by transferring the data into nodes and its relationships into edges. Typically built and used for in depth relationships in the data. Relationships are first class objects. Recommendations in eCommerce sites use graph technology in the back end (customers who purchased this product also purchased these etc).

Examples: AllegroGraph, ArrangoDB, Amazon Neptune.

Time Series Database:

Optimised for handling time series data. A sequence of data points recorded over time. Time is the single primary axis of the data model.

Examples: Cube, InfluxDB, TimescaleDB, Amazon TimeSeries

Modern application might need to use more than one database. If you take any typical retail (ecommerce) application, it will definitely use more than one database for different purposes as described below,

• RDBMS for the actual transaction data.
• Key value DB for product catalogue, shopping cart.
• Search database for product search.
• Cache database for leader boards, top rated products etc.
• Graph database for recommendations.

I hope this post provided the high level view of different types of databases available and the purpose behind the each option.

AWS Cloud Storage - Part 2

This post is continuation to my previous post on AWS Cloud Storage. In this post I will discuss about the other offerings/options that Amazon provides with respect to storage.

Amazon Elastic File System (EFS)
Amazon EFS delivers a simple, scalable, elastic, highly available and highly durable network file system as a service to EC2 instances.It supports Network File System versions 4 (NFSv4) and 4.1 (NFSv4.1), which makes it easy to migrate enterprise applications to AWS or build new ones.
Amazon EFS is designed to meet the needs of multi-threaded applications and applications that concurrently access data from multiple EC2 instances and that require substantial levels of aggregate throughput and input/output operations per second (IOPS). 

Amazon Elastic Block Storage(EBS)
Amazon Elastic Block Store (Amazon EBS) volumes provide durable block-level storage for use with EC2 instances.  Amazon EBS volumes are network-attached storage that persists independently from the running life of a single EC2 instance. After an EBS volume is attached to an EC2 instance, you can use the EBS volume like a physical hard drive, typically by formatting it with the file system of your choice and using the file I/O interface provided by the instance operating system. Most Amazon Machine Images (AMIs) are backed by Amazon EBS, and use an EBS volume to boot EC2 instances. You can also attach multiple EBS volumes to a single EC2 instance. Note, however, that any single EBS volume can be attached to only one EC2 instance at any time. EBS also provides the ability to create point-in-time snapshots of volumes, which are stored in Amazon S3.

Amazon EBS is meant for data that changes relatively frequently and needs to persist beyond the life of EC2 instance. Amazon EBS is well-suited for use as the primary storage for a database or file system, or for any application or instance (operating system) that requires direct access to raw block-level storage. Amazon EBS provides a range of options that allow you to optimize storage performance and cost for your workload. These options are divided into two major categories: solid-state drive (SSD)-backed storage for transactional workloads such as databases and boot volumes (performance depends primarily on IOPS) and hard disk drive (HDD)-backed storage for throughput-intensive workloads such as big data, data warehouse, and log processing (performance depends primarily on MB/s).

Amazon EC2 Instance Storage
Amazon EC2 instance store volumes (also called ephemeral drives) provide temporary block-level storage for many EC2 instance types.This storage consists of a preconfigured and pre-attached block of disk storage on the same Amazon Web Services – AWS Storage Services Overview Page physical server that hosts the EC2 instance for which the block provides storage. The amount of the disk storage provided varies by EC2 instance type. 

AWS Storage Gateway
Service that connects your on-premises to aws cloud. The service enables you to securely store data in the AWS Cloud for scalable and cost-effective storage. AWS Storage Gateway supports industry standard storage protocols that work with your existing applications. It provides low-latency performance by maintaining frequently accessed data on-premises while securely storing all of your data encrypted in Amazon S3 or Amazon Glacier. You can download the AWS Storage Gateway software appliance as a virtual machine (VM) image that you install on a host in your data center or as an EC2 instance.
Types

• File gate (NFS) - flat files in S3 (pdf, jpeg ,word etc)
• Volumes Gateway(iSCSI) - Block storage , virtual hard disk (will not be stored in S3)
• stored volumes - whole data
• cached volumes  - cached data
• Tape Gateway (VTL) - virtual tapes

AWS Snowball
AWS import/export to accelerate moving large amounts of data into and out of AWS.
Types

• snowball - peta bytes scale data transfer solution only storage
• Snowball edge - 100tb data with storage and compute (little aws data center)
• snowmobile - shipping container for hexa byte of data 100pb per snowmobile , data centre migration

AWS CloudFront
CloudFront is a content delivery service that speeds up the distribution of your website’s dynamic, static, and streaming content by making it available from a global network of edge locations. CDN is a system of distributed servers that deliver web pages and web content to user based on the geographic locations of the user, the origin of the web page and a content delivery server.

• Edge location - location where content will be cached.
• Origin - origin of all files that CDN will distribute (S3, EC2, ELB, route 53).
• Distribution - this it he name given the CDN which consists of collection of edge locations.
• Web distribution - for web sites.
• RTMP - used for media streaming.
• Edge location are not just read only , you can write to them too.
• Objects are cached for the life of TTL(time to live).
• you can clear the cached object but you will be charged.

AWS Cloud Storage

This post is continuation to my previous post about AWS introduction. I will try to explain each service that is described in the previous post in detail. In this post, I will discuss about storage cloud service of Amazon web services.

Amazon Simple Storage Service (S3)

A service that provides highly scalable and highly durable object storage in cloud. Here object storage means the architecture that manages data as objects, as opposed to other storage architectures like file systems and block storage. You can store and retrieve any amount of data, at any time from any where on the web through a simple web service interface.

S3 storage classes/tiers

• S3 - standard: For general purpose storage of frequently accessed data. Provides 99.99 availability, 11 9's durability designed to sustain the loss of 2 facilities concurrently.
• S3 - IA (Infrequent Access): For data that is accessed less frequently but requies rapid access when needed. Lower fee and retrieval fee - stored in multiple availability zones.
• S3 one zone IA : Lower cost than IA but stored in one AZ only.
• Glacier: For low cost archival data. Glacier provides three options for retrieval. Expedited ( retrieval in minutes), standard (retrieval in 3-5 hrs), bulk (retrieval in 5-12 hours).

S3 is for

• To store distributed files, static web content and media. S3 can server as an origin store for content delivery network (CDN), such as Amazon cloudfront. Eg: photo and video sharing sites.
• S3 can be used to host entire static websites.
• Backup and archival of data.

S3 is not for

• S3 is not meant for file system.
• Storing databases.
• Dynamic website hosting.

Pricing

You pay for the storage that you actually use. No minimum fee and no setup cost. S3 has three pricing components: storage (per GB per month), data transfer in or out (per GB per month) and requests (per thousand requests per month). For data transfer, there is transfer acceleration option which provides better performance than regular S3 transfer.

Cross Region Replication

Cross region replication allows objects to be copied across s3 buckets in different AWS regions asynchronously providing disaster recovery solutions for business continuity. 

Cross region replication needs versioning to be enabled on both buckets. You can change the class of the replication (S3 in source and S3-IA in destination). 

• Only new objects or the modified objects will be replicated (old ones will not be).
• Delete will replicate the delete marker.
• Deleting the delete marker will not be replicated.
• Deleting version of a file will not be replicated.
• Cannot replicate to multiple buckets.

Encryption
You can choose to encrypt the data using SSE-S3, SSE-C, SSE-KMS, or a client library. All four enable you to store sensitive data encrypted at rest in Amazon S3.
SSE-S3 provides a solution where amazon handles key management and key protection using multiple layers of security.
SSE-C enables you to leverage Amazon S3 to perform the encryption and decryption of your objects while retaining control of the keys used to encrypt the objects. 
SE-KMS enables you to use AWS Key Management Service to manage your encryption keys.
Client Library: Using an encryption client library, such as the amazon s3 encryption client, you retain control of the keys and complete the encryption and decryption of objects client-side using an encryption library of your choice.

Event Notifications
Amazon S3 event notifications can be sent in response to actions in amazon s3 like PUTs, POSTs, COPYs or DELETEs. Notificaiton messages can be sent through either Amazon SNS, Amason SQS or to AWS Lambda. Amazon S3 event notifications enable you to run workflows, send alerts, or perform other actions in response to changes in your objects stored in S3.

Points to remember

• S3 is object based.
• Simple key/value store (key - name of the file , value- data)
• Files can be from 0 bytes to 5Tb.
• Virtually unlimited storage.
• Files stored in buckets (like folder in cloud).
• Bucket name use universal namespace (must be unique globally).
• Read after write consistency for new objects.
• Eventual consistency for overwrite puts and deletes (some time to propagate and reflect).
• Large objects can be uploaded using multipart upload.
• Supports versioning. Versioning can only be enabled, it cannot be disabled (can be suspended).
• Normal delete will create the delete marker (which marks the object is deleted, the actual object will still be there).
• If you delete the delete marker then it will restore the object.
• Each object can be accessed through an unique http URL.
• You have to select a region while creating bucket.
• Transfer Acceleration utilises cloudfront to accelerate your uploads to S3. you upload to edge location and then transfer that file to S3. yo will get a distinct url to upload to.
• There are four mechanisms to control access to S3 resources: IAM (Identity and Access Management) policies, Access control lists, bucket policies and query string authentication.
• Amazon S3 Block Public Access is a new set of security controls that allows customers to enforce that S3 buckets and objects do not have public access. This setting overrides all other access level setting.
• To retrieve Amazon S3 data stored in Amazon Glacier, initiate a retrieval request using the Amazon S3 APIs or the Amazon S3 Management Console. The retrieval request creates a temporary copy of your data in the S3 RRS or S3 Standard-IA storage class while leaving the archived data intact in Amazon Glacier. You can specify the amount of time in days for which the temporary copy is stored in S3. You can then access your temporary copy from S3 through an Amazon S3 GET request on the archived object.
• Using IPv6 support for Amazon S3, applications can connect to Amazon S3 without the need for any IPv6 to IPv4 translation software or systems. You can get started by pointing your application to Amazon S3’s new “dual-stack” endpoint, which supports access over both IPv4 and IPv6.

Configuring SSL in JCS/SOAcs Server with PFX certificate file

PFX file is the file generated by windows ISS which contains certificate and private key of the certificate. The steps to apply this certificate on java cloud service or soa cloud service are described below.

1. Export the certificate with its private key from IIS as .pfx file.
2. Move the .pfx file to the bin folder of keytool in oracle soacs VM
• Connect to VM through winscp and move the file using opc user.
• Connect to the VM through putty and change the permissions (chmod) and change owner (chown).
• Move the file to /u01 (or some other location of your choice)
3. Now we need to convert the .pfx file to jks. Use the below command to achieve this,

$JAVA_HOME/bin/keytool -v -importkeystore -srckeystore <<pfx_file>> -srcstoretype PKCS12 -destkeystore <<destincation_jks>> -deststoretype JKS This will ask for destination password (you have to set new password) Source password: Enter the password of pfx private key

• View the content of the newly created JKS to obtain the original alias of private key

$JAVA_HOME/bin/ keytool -list -v -keystore <<jks_keystore_created_above>> Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry Alias name: <<alias will be printed here>>

• Change the original alias to a simpler alias

$JAVA_HOME/bin/keytool -keystore <<new_keystore.jsk>> -storepass <<password>> -changealias -alias <<original_alias>> -keypass <<orinial pfx password>> -destalias <<new _alias>>

• Now the new JKS file is ready which needs to be used as weblogic identity store.
• Login to weblogic admin console, Lock & edit.
• Navigate to environment à servers.
• Select admin server, click on keystores tab
• Change the keystore to “Custom Identity and Java Trust Store”

Custom Identity Store = <<path of new JKS>> Custom Identity Keystore Type = JKS Custom Identity Keystore Passphrase = <<keystore password>> Confirm Custom Identity Keystore Passphrase = << keystore password >>

• Go to SSL tab

Private Key Alias= <<Alias that you created in step 5>> Private Key Passphrase = <<password of private key>> Confirm Private Key Passphrase = <<password of private key>>

• Save & Release Configuration.
• Restart the SSL (from Environment à Servers, control tab)
• Repeat the above five steps for each managed server.
• Restart the SSL of each server. Now you can access your server/apps over https.

Certificate error while invoking REST adapter in Oracle SOA 12c

You might get into certificate error (valid certificate path not found) while invoking a rest service through rest adapter in Oracle SOA even though you install the certificates using keytool.

To overcome this issue, 

• Login to EM console
• Navigate to weblogic Domain --> Security --> Keystore
• Select the System (row) --> select trust 
• Click on Manage
• Import the root certificate
• Restart the server and it should work.

Merge query & Exposing as a Rest Service in Schema as a Service

In this post, I will describing the steps required to write a merge/upsert query (insert or update) query and expose it as a rest service in Oracle DBCS Schema as a service.

Merge query
Use merge statement to conditionally insert or update data depending on its presence, a process known as upsert. The Merge statement selects data from one or more source tables and updates or inserts into the target table. The merge statement allows you to specify a condition to determine whether to update data from or insert into the target table.

Syntax:

MERGE INTO target_table 
USING source_table 
ON search_condition
    WHEN MATCHED THEN
        UPDATE SET col1 = value1, col2 = value2,...
        WHERE <update_condition>
    WHEN NOT MATCHED THEN
        INSERT (col1,col2,...)
        values(value1,value2,...)

        WHERE <insert_condition>;

In Oracle Schema as a Service

Oracle schema as a service is a database schema which can be accessed only through browser based application called APEX. There are only two ways to perform data exchange with with schema as a service.

• Accessing by coding applications (in ADF or JEE) in associated JCS SX (SaaS Extensions) service.
• Exposing queries as RESTful web services.

We will see the second option with merge query,

• Login to you DBCS Schema as a service.
• Navigate to SQL Workshop --> Restful Services
• Click on Create and provide all mandatory fields as per your database object (URI prefix is the base of your URI .
• Click on "Create Template" to create the URI template for your respective object.
• Create Handler for the upsert operation. You can select either "POST" or "PUT" as your method. Since we are talking about upsert (merge) operation it is good practice to select PUT.
• Enter "application/json" in MIME Types Allowed field.
• Enter your PL sql query in source field.
• Merge query requires the source from another table, but as always we have  dummy table dual from which we can fetch our input.
• I am giving the example query to merge into department table below,

begin

merge into dept d 

using (select :deptNo deptNo, :dName dname,:loc loc from dual) s

on (d.deptNo = s.deptNo)

 when matched then update set dname = s.dname, loc=s.loc

  when not matched then insert (deptNo, dname, loc) 

    values (s.deptNo,s.dname, s.loc);

   end;

• You can frame the URL of this REST service by appending the URI's that you had entered for module and template to the base URL of your schema as a service instance.
• You can pass the input in the body of the rest service input if json format as mentioned below,

{

"deptNo":"72",

"dName":"Marketing",

"loc":"Texas"

}

The above steps allows you to expose your merge query as rest service and if you want the service to be secure then select "yes" for Requires Secure Access field(which mandates access over ssl) and to attach authentication please follow the steps mentioned in the oracle documentation for adding Restful Service Privileges.