AWS Cloud Storage - Part 2

This post is continuation to my previous post on AWS Cloud Storage. In this post I will discuss about the other offerings/options that Amazon provides with respect to storage.

Amazon Elastic File System (EFS)
Amazon EFS delivers a simple, scalable, elastic, highly available and highly durable network file system as a service to EC2 instances.It supports Network File System versions 4 (NFSv4) and 4.1 (NFSv4.1), which makes it easy to migrate enterprise applications to AWS or build new ones.
Amazon EFS is designed to meet the needs of multi-threaded applications and applications that concurrently access data from multiple EC2 instances and that require substantial levels of aggregate throughput and input/output operations per second (IOPS). 

Amazon Elastic Block Storage(EBS)
Amazon Elastic Block Store (Amazon EBS) volumes provide durable block-level storage for use with EC2 instances.  Amazon EBS volumes are network-attached storage that persists independently from the running life of a single EC2 instance. After an EBS volume is attached to an EC2 instance, you can use the EBS volume like a physical hard drive, typically by formatting it with the file system of your choice and using the file I/O interface provided by the instance operating system. Most Amazon Machine Images (AMIs) are backed by Amazon EBS, and use an EBS volume to boot EC2 instances. You can also attach multiple EBS volumes to a single EC2 instance. Note, however, that any single EBS volume can be attached to only one EC2 instance at any time. EBS also provides the ability to create point-in-time snapshots of volumes, which are stored in Amazon S3.

Amazon EBS is meant for data that changes relatively frequently and needs to persist beyond the life of EC2 instance. Amazon EBS is well-suited for use as the primary storage for a database or file system, or for any application or instance (operating system) that requires direct access to raw block-level storage. Amazon EBS provides a range of options that allow you to optimize storage performance and cost for your workload. These options are divided into two major categories: solid-state drive (SSD)-backed storage for transactional workloads such as databases and boot volumes (performance depends primarily on IOPS) and hard disk drive (HDD)-backed storage for throughput-intensive workloads such as big data, data warehouse, and log processing (performance depends primarily on MB/s).

Amazon EC2 Instance Storage
Amazon EC2 instance store volumes (also called ephemeral drives) provide temporary block-level storage for many EC2 instance types.This storage consists of a preconfigured and pre-attached block of disk storage on the same Amazon Web Services – AWS Storage Services Overview Page physical server that hosts the EC2 instance for which the block provides storage. The amount of the disk storage provided varies by EC2 instance type. 

AWS Storage Gateway
Service that connects your on-premises to aws cloud. The service enables you to securely store data in the AWS Cloud for scalable and cost-effective storage. AWS Storage Gateway supports industry standard storage protocols that work with your existing applications. It provides low-latency performance by maintaining frequently accessed data on-premises while securely storing all of your data encrypted in Amazon S3 or Amazon Glacier. You can download the AWS Storage Gateway software appliance as a virtual machine (VM) image that you install on a host in your data center or as an EC2 instance.
Types

• File gate (NFS) - flat files in S3 (pdf, jpeg ,word etc)
• Volumes Gateway(iSCSI) - Block storage , virtual hard disk (will not be stored in S3)
• stored volumes - whole data
• cached volumes  - cached data
• Tape Gateway (VTL) - virtual tapes

AWS Snowball
AWS import/export to accelerate moving large amounts of data into and out of AWS.
Types

• snowball - peta bytes scale data transfer solution only storage
• Snowball edge - 100tb data with storage and compute (little aws data center)
• snowmobile - shipping container for hexa byte of data 100pb per snowmobile , data centre migration

AWS CloudFront
CloudFront is a content delivery service that speeds up the distribution of your website’s dynamic, static, and streaming content by making it available from a global network of edge locations. CDN is a system of distributed servers that deliver web pages and web content to user based on the geographic locations of the user, the origin of the web page and a content delivery server.

• Edge location - location where content will be cached.
• Origin - origin of all files that CDN will distribute (S3, EC2, ELB, route 53).
• Distribution - this it he name given the CDN which consists of collection of edge locations.
• Web distribution - for web sites.
• RTMP - used for media streaming.
• Edge location are not just read only , you can write to them too.
• Objects are cached for the life of TTL(time to live).
• you can clear the cached object but you will be charged.

AWS Cloud Storage

This post is continuation to my previous post about AWS introduction. I will try to explain each service that is described in the previous post in detail. In this post, I will discuss about storage cloud service of Amazon web services.

Amazon Simple Storage Service (S3)

A service that provides highly scalable and highly durable object storage in cloud. Here object storage means the architecture that manages data as objects, as opposed to other storage architectures like file systems and block storage. You can store and retrieve any amount of data, at any time from any where on the web through a simple web service interface.

S3 storage classes/tiers

• S3 - standard: For general purpose storage of frequently accessed data. Provides 99.99 availability, 11 9's durability designed to sustain the loss of 2 facilities concurrently.
• S3 - IA (Infrequent Access): For data that is accessed less frequently but requies rapid access when needed. Lower fee and retrieval fee - stored in multiple availability zones.
• S3 one zone IA : Lower cost than IA but stored in one AZ only.
• Glacier: For low cost archival data. Glacier provides three options for retrieval. Expedited ( retrieval in minutes), standard (retrieval in 3-5 hrs), bulk (retrieval in 5-12 hours).

S3 is for

• To store distributed files, static web content and media. S3 can server as an origin store for content delivery network (CDN), such as Amazon cloudfront. Eg: photo and video sharing sites.
• S3 can be used to host entire static websites.
• Backup and archival of data.

S3 is not for

• S3 is not meant for file system.
• Storing databases.
• Dynamic website hosting.

Pricing

You pay for the storage that you actually use. No minimum fee and no setup cost. S3 has three pricing components: storage (per GB per month), data transfer in or out (per GB per month) and requests (per thousand requests per month). For data transfer, there is transfer acceleration option which provides better performance than regular S3 transfer.

Cross Region Replication

Cross region replication allows objects to be copied across s3 buckets in different AWS regions asynchronously providing disaster recovery solutions for business continuity. 

Cross region replication needs versioning to be enabled on both buckets. You can change the class of the replication (S3 in source and S3-IA in destination). 

• Only new objects or the modified objects will be replicated (old ones will not be).
• Delete will replicate the delete marker.
• Deleting the delete marker will not be replicated.
• Deleting version of a file will not be replicated.
• Cannot replicate to multiple buckets.

Encryption
You can choose to encrypt the data using SSE-S3, SSE-C, SSE-KMS, or a client library. All four enable you to store sensitive data encrypted at rest in Amazon S3.
SSE-S3 provides a solution where amazon handles key management and key protection using multiple layers of security.
SSE-C enables you to leverage Amazon S3 to perform the encryption and decryption of your objects while retaining control of the keys used to encrypt the objects. 
SE-KMS enables you to use AWS Key Management Service to manage your encryption keys.
Client Library: Using an encryption client library, such as the amazon s3 encryption client, you retain control of the keys and complete the encryption and decryption of objects client-side using an encryption library of your choice.

Event Notifications
Amazon S3 event notifications can be sent in response to actions in amazon s3 like PUTs, POSTs, COPYs or DELETEs. Notificaiton messages can be sent through either Amazon SNS, Amason SQS or to AWS Lambda. Amazon S3 event notifications enable you to run workflows, send alerts, or perform other actions in response to changes in your objects stored in S3.

Points to remember

• S3 is object based.
• Simple key/value store (key - name of the file , value- data)
• Files can be from 0 bytes to 5Tb.
• Virtually unlimited storage.
• Files stored in buckets (like folder in cloud).
• Bucket name use universal namespace (must be unique globally).
• Read after write consistency for new objects.
• Eventual consistency for overwrite puts and deletes (some time to propagate and reflect).
• Large objects can be uploaded using multipart upload.
• Supports versioning. Versioning can only be enabled, it cannot be disabled (can be suspended).
• Normal delete will create the delete marker (which marks the object is deleted, the actual object will still be there).
• If you delete the delete marker then it will restore the object.
• Each object can be accessed through an unique http URL.
• You have to select a region while creating bucket.
• Transfer Acceleration utilises cloudfront to accelerate your uploads to S3. you upload to edge location and then transfer that file to S3. yo will get a distinct url to upload to.
• There are four mechanisms to control access to S3 resources: IAM (Identity and Access Management) policies, Access control lists, bucket policies and query string authentication.
• Amazon S3 Block Public Access is a new set of security controls that allows customers to enforce that S3 buckets and objects do not have public access. This setting overrides all other access level setting.
• To retrieve Amazon S3 data stored in Amazon Glacier, initiate a retrieval request using the Amazon S3 APIs or the Amazon S3 Management Console. The retrieval request creates a temporary copy of your data in the S3 RRS or S3 Standard-IA storage class while leaving the archived data intact in Amazon Glacier. You can specify the amount of time in days for which the temporary copy is stored in S3. You can then access your temporary copy from S3 through an Amazon S3 GET request on the archived object.
• Using IPv6 support for Amazon S3, applications can connect to Amazon S3 without the need for any IPv6 to IPv4 translation software or systems. You can get started by pointing your application to Amazon S3’s new “dual-stack” endpoint, which supports access over both IPv4 and IPv6.

Configuring SSL in JCS/SOAcs Server with PFX certificate file

PFX file is the file generated by windows ISS which contains certificate and private key of the certificate. The steps to apply this certificate on java cloud service or soa cloud service are described below.

1. Export the certificate with its private key from IIS as .pfx file.
2. Move the .pfx file to the bin folder of keytool in oracle soacs VM
• Connect to VM through winscp and move the file using opc user.
• Connect to the VM through putty and change the permissions (chmod) and change owner (chown).
• Move the file to /u01 (or some other location of your choice)
3. Now we need to convert the .pfx file to jks. Use the below command to achieve this,

$JAVA_HOME/bin/keytool -v -importkeystore -srckeystore <<pfx_file>> -srcstoretype PKCS12 -destkeystore <<destincation_jks>> -deststoretype JKS This will ask for destination password (you have to set new password) Source password: Enter the password of pfx private key

• View the content of the newly created JKS to obtain the original alias of private key

$JAVA_HOME/bin/ keytool -list -v -keystore <<jks_keystore_created_above>> Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry Alias name: <<alias will be printed here>>

• Change the original alias to a simpler alias

$JAVA_HOME/bin/keytool -keystore <<new_keystore.jsk>> -storepass <<password>> -changealias -alias <<original_alias>> -keypass <<orinial pfx password>> -destalias <<new _alias>>

• Now the new JKS file is ready which needs to be used as weblogic identity store.
• Login to weblogic admin console, Lock & edit.
• Navigate to environment à servers.
• Select admin server, click on keystores tab
• Change the keystore to “Custom Identity and Java Trust Store”

Custom Identity Store = <<path of new JKS>> Custom Identity Keystore Type = JKS Custom Identity Keystore Passphrase = <<keystore password>> Confirm Custom Identity Keystore Passphrase = << keystore password >>

• Go to SSL tab

Private Key Alias= <<Alias that you created in step 5>> Private Key Passphrase = <<password of private key>> Confirm Private Key Passphrase = <<password of private key>>

• Save & Release Configuration.
• Restart the SSL (from Environment à Servers, control tab)
• Repeat the above five steps for each managed server.
• Restart the SSL of each server. Now you can access your server/apps over https.

Certificate error while invoking REST adapter in Oracle SOA 12c

You might get into certificate error (valid certificate path not found) while invoking a rest service through rest adapter in Oracle SOA even though you install the certificates using keytool.

To overcome this issue, 

• Login to EM console
• Navigate to weblogic Domain --> Security --> Keystore
• Select the System (row) --> select trust 
• Click on Manage
• Import the root certificate
• Restart the server and it should work.

Merge query & Exposing as a Rest Service in Schema as a Service

In this post, I will describing the steps required to write a merge/upsert query (insert or update) query and expose it as a rest service in Oracle DBCS Schema as a service.

Merge query
Use merge statement to conditionally insert or update data depending on its presence, a process known as upsert. The Merge statement selects data from one or more source tables and updates or inserts into the target table. The merge statement allows you to specify a condition to determine whether to update data from or insert into the target table.

Syntax:

MERGE INTO target_table 
USING source_table 
ON search_condition
    WHEN MATCHED THEN
        UPDATE SET col1 = value1, col2 = value2,...
        WHERE <update_condition>
    WHEN NOT MATCHED THEN
        INSERT (col1,col2,...)
        values(value1,value2,...)

        WHERE <insert_condition>;

In Oracle Schema as a Service

Oracle schema as a service is a database schema which can be accessed only through browser based application called APEX. There are only two ways to perform data exchange with with schema as a service.

• Accessing by coding applications (in ADF or JEE) in associated JCS SX (SaaS Extensions) service.
• Exposing queries as RESTful web services.

We will see the second option with merge query,

• Login to you DBCS Schema as a service.
• Navigate to SQL Workshop --> Restful Services
• Click on Create and provide all mandatory fields as per your database object (URI prefix is the base of your URI .
• Click on "Create Template" to create the URI template for your respective object.
• Create Handler for the upsert operation. You can select either "POST" or "PUT" as your method. Since we are talking about upsert (merge) operation it is good practice to select PUT.
• Enter "application/json" in MIME Types Allowed field.
• Enter your PL sql query in source field.
• Merge query requires the source from another table, but as always we have  dummy table dual from which we can fetch our input.
• I am giving the example query to merge into department table below,

begin

merge into dept d 

using (select :deptNo deptNo, :dName dname,:loc loc from dual) s

on (d.deptNo = s.deptNo)

 when matched then update set dname = s.dname, loc=s.loc

  when not matched then insert (deptNo, dname, loc) 

    values (s.deptNo,s.dname, s.loc);

   end;

• You can frame the URL of this REST service by appending the URI's that you had entered for module and template to the base URL of your schema as a service instance.
• You can pass the input in the body of the rest service input if json format as mentioned below,

{

"deptNo":"72",

"dName":"Marketing",

"loc":"Texas"

}

The above steps allows you to expose your merge query as rest service and if you want the service to be secure then select "yes" for Requires Secure Access field(which mandates access over ssl) and to attach authentication please follow the steps mentioned in the oracle documentation for adding Restful Service Privileges.

AWS Introduction - Part 1

I am going through a self learning course on Amazon Web Services (AWS) platform. I will to jotting down my notes in series of posts on different services that AWS offers. Lets get started.
This will be the introduction part 1 where I will be covering the basic concepts of AWS and all the services a very high level.

AWS Global Infrastructure

Region: A region is a physical geographic location which contains two (or more) availability zones.

Availability Zone (AZ): Availability zone is simply a data center within a region. Each region consists of two (or more) availability zones.

AWS Edge Location: Edge locations are end points for AWS which are used for caching the content.

AWS global infrastructure is the base for all AWS services. All the AWS services are built on top the global infrastructure. 

Building ADF applications using Developer cloud service

In this post I will be demonstrating the steps required to build ADF applications using Oracle Developer Cloud Service. I will be using ant to do the build activity, the steps are more or less same even if you are planning to use maven.

• Create a project and repository in developer cloud service.
• Check-in all your project code to GIT repository of developer cloud service.
• Generate the build files (build.xml and build.properties) if not present in your project.
• Jdeveloper generated build files will work when build in your local environment, when you need to build it in developer cloud service, you need to tweak the files a bit. 

build.properties file

oracle.commons=../../../../oracle_common/
install.dir=../../../..
oracle.home=${env.ORACLE_HOME_12C3}
oracle.jdeveloper.workspace.path=${env.WORKSPACE}/SampleBuildApp.jws
middleware.home=${env.MIDDLEWARE_HOME_12C3}
workspace=${env.WORKSPACE}
oracle.jdeveloper.ant.library=${env.ORACLE_HOME_12C3}/jdev/lib/ant-jdeveloper.jar
oracle.jdeveloper.deploy.dir=${env.WORKSPACE}/ViewController/deploy
oracle.jdeveloper.ojdeploy.path=${oracle.home}/jdev/bin/ojdeploy
javac.nowarn=off
oracle.jdeveloper.project.name=ViewController
oracle.jdeveloper.deploy.outputfile=${env.WORKSPACE}/deploy/${profile.name}
output.dir=classes
javac.deprecation=off
oracle.jdeveloper.deploy.profile.name=*
javac.debug=on

• Change SampleBuildApp.jws to your application name.

build.xml file

<?xml version="1.0" encoding="windows-1252" ?>

<project xmlns="antlib:org.apache.tools.ant" name="SampleBuildApp" basedir=".">

  <property environment="env" />

  <property file="build.properties"/>

  <target name="deploy" description="Deploy JDeveloper profiles">

    <taskdef name="ojdeploy" classname="oracle.jdeveloper.deploy.ant.OJDeployAntTask" uri="oraclelib:OJDeployAntTask"

             classpath="${oracle.jdeveloper.ant.library}"/>

    <ora:ojdeploy xmlns:ora="oraclelib:OJDeployAntTask" executable="${oracle.jdeveloper.ojdeploy.path}"

                  ora:buildscript="${oracle.jdeveloper.deploy.dir}/ojdeploy-build.xml"

                  ora:statuslog="${oracle.jdeveloper.deploy.dir}/ojdeploy-statuslog.xml">

      <ora:deploy>

        <ora:parameter name="workspace" value="${oracle.jdeveloper.workspace.path}"/>

        <ora:parameter name="profile" value="${oracle.jdeveloper.deploy.profile.name}"/>

        <ora:parameter name="nocompile" value="false"/>

        <ora:parameter name="outputfile" value="${oracle.jdeveloper.deploy.outputfile}"/>

      </ora:deploy>

    </ora:ojdeploy>

  </target>

</project>

• Don't forget to add the property environment (highlighted in red).
• Check-in these files to developer cloud service.
• Create a new build job by selecting the application git repository as source and deploy as build task and in the post build steps select Archive the artifact option and enter deploy/*.ear in files to archive field.

.

• Run the build operation by clicking build now button.
• Once the build is completed, you can find the .ear file under deploy folder of artifacts.

• You can take this file and deploy in you JCS or on-premises environment.

Oracle Intelligent bot platform - Introduction

I am going through an online course from oracle on their Intelligent bot platform of. I will be jotting down the key concepts of this course and platform so that it will act as a quick reference guide for anyone who is starting to work on this technology and for people (like me) who want a quick reference. Let’s get started,

What are Intelligent bots??

A computer program that you can interact with in a conversational manner. The user experience of intelligent bot is to simulate a human being as a conversational partner. There are majorly two types of bots, the simpler ones scans for the keywords in the user input and more sophisticated ones which uses natural language processing (NLP) systems (to understand natural human conversations) .

Building blocks and terminology of Oracle intelligent bots

•        Intents  
•        Utterances
•        Entity
•        Intelligence
•        Flow

Let us understand each one of them,

Intent:  Intent is like an use case a bot supports, any unit of work a bot supposed to handle. The key feature of any intelligent bot is “ A bot should be able to resolve the input to the same intention of the user”. Example: Check balance, transfer money etc.

Utterance: The way to connect user input to intents. You need to train the bot by giving utterances. Example for check balance can be what is my balance, tell my balance, how much money in my savings account etc

Entity: A variable element of an intent that you want to parameterize to allow input of different values. Bot resolves intent and then entity. Entity example for check balance intent will be savings, current and credit card.

Oracle intelligent bot has built in entities for common entities like date, currency, address etc.

Intelligence: Heart of any chat bot platform. The ability to understand users natural language conversations using natural language processing (NLP) capability.

Flow: Any Conversion will always have a flow to handle the user interactions.