LFTP is a command-line program for several file transfer protocols. LFTP is designed for Unix and linux type of operating systems and it is distributed under GNU General Public License, so we are free to install and use.
LFTP can transfer files via FTP, FTPS, HTTP, HTTPS, FISH, SFTP, Bit Torrent and FTP over HTTP proxy. It also supports the File eXchange Protocol (FXP), which allows the client to transfer files from one remote FTP server to another.
The features of LFTP's are transfer queues, segmented file transfer, resuming partial downloads, mirror directories, bandwidth throttling, and recursive copying of file directories. The client can be used interactively or automated with scripts.
As part of my job, we deal with pulling large number of files from remote systems as and when the data is available, using java based sftp/ftp pull using JSCH library. This is also good for most of the scenarios but for every other different data collection we had to write bunch of code on top of the common JSCH based framework to collect data, mainly because of the directly structures and number of files we need collect are way different from one and other.
One of the cool future which caught my eye and perfect solution for the above mentioned problem is LFTP mirror.
In this Blog, I would like to discuss the overview about Karate Framework and internals of Karate will be discussed in the next blog post.
I am sure many people would have used Cucumber for some of the Test automation or Developing BDD style step execution process.
I recently got an opportunity to work in Functional Test automation for API in my project. we discussed internally as a team and decided to use Cucumber with Rest Assured for automating the functional test cases. we informed the same to our project director who is extremely knowledgeable and a great professor, so we were excited to hear his feed on our choice.
Immediately he pitched in and suggested karate framework which is new to me . My first feeling was, i like watching Karate but how does Karate fight help testing the API 😃
Immediately i started exploring Karate and its capabilities and Found that It is a great framework and made developers and testers life so easy in automating test scenarios, Thank you Peter (Director). and yes of-course it will have little bit of learning curve and i am sure one would not be disappointed learning it.
What is Functional Test Automation ?
Automated Testing is a complex subject in Software Development and usually not very prioritized, but once we start enhancing the projects and when they become big enough to manage, we will right away get to know the importance of Automation.
Testing Automation would save numerous hours of Testers and Developers time. Automating the Functional Test will help simplify the integration and regression testing in agile environment where many small releases going in by assuring the quality of product.
what is Cucumber Framework ?
Functional Testing is more meaningful when the platform allows all the stake holds starting from users, managers, testers and developers to participate in documenting the Functionalities of the API or a product. Using the same document for Testing functionality after completing the development. This requires a common framework to write the test scenarios in ubiquitous language.
Cucumber helps documenting Test scenarios in Behavior Driven Development(BDD) style which uses the natural language parser named Gherkin. we can write Gherkin steps in many human readable languages like English, French etc.
Cucumber has Three step process as pictured below :
step1: Parser the Gherkin step.
step2: call the function implemented for the step.
step 3: manipulate the end system and return the results.
An Example explaining the cucumber process :
Feature: Cell Site information API Functionality Testing
Scenario: Compare Cell Site information Given api end point url And authentication details And urlpath parameters When request method is GET Then check the returned status is 200 And compare the expected response with actual response
There are many tools and frameworks available to automate different phases of SDLC using cucumber. Two popular frameworks for API Testing automation are Karate and Rest Assured.
Using Rest Assured Developers/Testers need to write Step definition in Domain Specific language for each of these steps to perform some action in the system. This would take more time to implement when we have many API endpoints in a bigger system. This can be eliminated using Karate.
What is Karate ?
Karate is an open source framework used for automating API Testing, Unit Test mocks, performance-testing even UI testing and developed to work with BDD style Cucumber framework.
Advantages of using Karate in API Functional Testing automation :
One framework for automating API Testing, Unit Test mocks, performance-testing even UI testing
Eliminates Cucumber Step definition coding effort, so it could save lot time to develop and automate.
Powerful JSON & XML match and assertion are built in, which makes us to work with data and configuration values in creating Test Scenarios with feasibility.
Easy to work with JavaScript and Java functions, we can make use of this functionality when we need to write a complex logic to full fill the Test scenarios.
Parallel test execution, enable the test scenarios to run in parallel which would be very useful when running automation in CI/CD pipeline.
Scenario Outline: : API GetResponse Status Scenario Given path '' * def pathparams = * if (pathparams.startDate != null) pathparams.startDate = yesterday * if (pathparams.endDate != null) pathparams.endDate = yesterday And params pathparams When method GET Then status * match each response.result[*] == expectedresult.result[0]
Many API's are developed to exchange data between applications and microservices. So it is highly important to adopt and enhance Karate framework for Functional Test automation. This would help find the issues related to functionality with less effort in the early stages of SDLC and provide very good experience for the API users.
I am a Software Engineer by profession and writing blogs in technology is my interest.
you might be wondering why is this person talking about Yoga and Meditation here. how is it related to Software or technology. we will discuss this in a while.
Yoga and Meditation are buzz words in many countries. i am not going to debate on where it born and who practiced it first. But i would like to discuss about the essence of these two buzz words.
whether we know it or not each one of us would be practicing both, one way or the other way.
is it true ? every one from east to west, south to north practicing yoga and meditation ?
The answer is YES !! may be in different forms.
i would try to explain what is Yoga and Meditation Programmatically, which might give my inner feeling for the above answer.
Human body is made up of different Mechanical and Electrical parts governed by Software Program running in Brain.
In my view Mechanical parts are bones, muscle and joints
Electrical parts are blood and fluids (current), Electrical machinery are heart and other internal organs which are dependent on blood and nerves are electric wires to carry current.
Electronic control board contain set of micro controllers running a Software program is brain, and it is responsible for sending instructions (digital codes) to machinery to run and move different mechanical parts.
Yoga is a practice to send set of instructions to brain to move certain parts of our Electrical body parts. This would eventually take out toxins deposited over a period of time in different electrical and mechanical parts and keep them healthy.
Meditation is a practice of cleaning the unwanted code and formatting the code written in Software Program running in brain.
let me explain in detail.
Every one in this universe born with simple program like below.
import milkyway.sun.earth.*;
public class MyNameUUID implements UniversalHumanInterface{
public static String RESULT = "COMPLETE"
public static void main(String[] args){
sendResult();
}
@Override
public String sendResult(){
return RESULT;
}
}
We(humans) would not be happy with looking at simple code like above, so as we grow we build complex logic by exposing to different relations, challenges, happy, anger, ego and lust.
each one of it forms functions, if/else conditions, loops, static variables, temporary local variables, and also build entirely different class and make relation between them.
import milkyway.sun.earth.*;
public class MyNameUUID implements UniversalHumanInterface{
public static String RESULT = "COMPLETE";
Relatives myrelatives;
public static void main(String[] args){
if ( work().more ){
RESULT = marriage();
RESULT = kids();
}else{
RESULT = work(like);
}
}
@Override
public String sendResult(){
return RESULT;
}
Money work(){...}
Pride work(like){...}
Happy marriage(){...}
Happy kids(){...}
Happy myrelatives.helpall();
Anger myrelatives.leavetherelation();
}
class Relatives {
void buildstrongrelation(){...}
void givemoney(){...}
void helpall(){...}
void leavetherelation(){...}
}
This chain of complex logic goes on and on every minute. and at one point of time it is very hard to manage and understand. So we would like to revisit the code and see what and why we have written this code.
some people wanted to do this cleanup early age ( 18 - 25 ) and some people do not wanted to revisit the code and happy being writing the code on and on..
majority of the people would realize to revisit the code in their mid age ( 45 - 60 ).
When we sit back and start reading the code we have written from many years, at first we may not be able to understand why we have written this code. we should keep looking at it and see what is important and what is not, clear the unwanted code, well format it and make only essential relations between the class. This cleaning activity is called Meditation.
Meditation process is ones choice, we could ignore putting extra effort to cleaning the code, we could clear portion of the code and leave the reset or we can completely format the code, This may take years because it all depends on ones effort to understand the code and make effective decision on what to keep and what to leave.
once we format the code in such a way that we can understand it at any point in time, we would be able to add only good and necessary code which we may write in future.
if we further simplifying code we end up having a class like below.
import milkyway.sun.earth.*;
public class MyNameUUID implements UniversalHumanInterface{
public static String RESULT = "COMPLETE";
public static void main(String[] args){
sendResult();
}
@Override
public String sendResult(){
return RESULT;
}
}
Exactly, we would not have anything in individuals class except what we had when we born. This is called Realization
some people may try to understand the code written on the other side of the UniversalHumanInterface.
some people may not be able to understand the code which they have written so they would seek help from others who were able to understand their code and go in their path.
In reality one does not need any ones help to understand their SELF written code, we need to keep practice to understand by our self. we would be able to better judge on what is required and what is not and decide accordingly.
This is the essence of Yoga and Meditation in my view 😃 ⏳
Both SSL (Secure Sockets Layer) and TLS(Transport Layer Security) are cryptographic protocols that provide security over a network. They are widely used across different applications. SSL had a number of vulnerabilities, and hence, TLS was introduced as an updated version of SSL in the year 1999. SSL has been deprecated by the Internet Engineering Task Force (IETF) in 2015. Therefore, you should have TLS protocol enabled for your applications. You will still see a lot of SSL terminology being used across many organizations until people get accustomed to TLS.
TLS/SSL enabled servers will have two main additional duties
Trust: Server application who serves the information to the "clients" needs to obtain trust by the clients. One way to achieve this by attaining TLS certificate from a globally trusted Certificate Authority, whom client has trust in.
Secure data: data exchange between client and server needs to go through a secured channel by "encrypting" the messages while it is transit.
Now we know the purpose of TLS/SSL, lets try to understand "How" this Trust and Secure data transfer happens between client and server.
Before we jump on to how TLS certificate works, lets understand what is Symmetric and Asymmetric Encryption. This would help understand the TLS completely.
symmetric Encryption: (data encryption and decryption happens with a common "secret_key"): In this case Client and Server holds the same "secret_key" to encrypt and decrypt the information. This is light weight and fast, since the size of the key is small, we can transfer more data with this approach. AES(Advanced Encryption Standard) is one of the common algorithms used for symmetric key encryption/decryption.
down side of this mechanism is, for the first time client has to send the "secret_key" to server safely to avoid Man In the Middle(MiM) attacks. which is not easy.
Asymmetric Encryption: (Client Encrypt with Public key and server decrypt with private key): In this case server generates a special private and public key pair, any messages encrypted with public key can only be decrypted with related public key.
public key would be shared to the clients and private key will be put in secret place at server side. Client would use public key to encrypt the information and server uses related private key to decrypt the information. since the size of the public key is more we would not be able to send large messages hence slow. RSA is one of the popular algorithm used to create Asymmetric keys
But this mechanism has a down side too that only client can send information to server securely, but server can not talk to the client.
There is a technique used to share symmetric key to the server for the first time, by using Asymmetric key approach. once the symmetric key reached the server safely, all the subsequent messages could be transferred using the symmetric key which is light weight and fast.
Steps involved in getting the Certificate signed by CA.
Server application (assume it is a "bank") request Intermediate Certificate Authority (ICA) by providing company name, address, email, domain and other details. This whole process may take 24 to 72 hours. Root CA will validate the below details before it provides a certificate to the requester.
Note: Intermediate CA is root CA authorized digital Certificate provider. certificate cost varies in attaining and managing the certificate between root CA and Intermediate CA.
verifying the legal, physical and operational existence of the entity
verifying that the identity of the entity matches official records
verifying that the entity has exclusive right to use the domain specified in the EV SSL Certificate
and verifying that the entity has properly authorized the issuance of the EV SSL Certificate.
Now server has CA provided certificate document, and it can be shared with the client to acquire confidence and have business with them. But how does server safely ship this certificate to the client browser or application. what if the document is hacked and corrupted by the MiM in transit. after all it is just a document with server details and authenticity.
The solution to this is "Digital signature"
Digital Signature is a private key encrypted value of the Cryptographic hash of the CA provided certificate document.
There are few Mathematical Hash algorithms to generate a Cryptographic Hash value from any kind of data. few of them are
Below are the Properties of the Cryptographic Hashing algorithm (lets take SHA256 as an example)
for Any kind of input value (text, file, mp4, jpg, pdf) it must generate 64 char (256 bit) unique value.
No Two hash values should be same for any two different inputs
hash value should be same for the same input, for any number of times we generate.
No one should NOT be able to recover the original input value from the hash value.
Now server can generate a hash out of the actual certificate file and encrypt the Cryptographic hash value using the private key to generate digital signature( aka cypher text) and send it to client.
On the client side two steps would be performed.
he can decrypt the digital signature using the public key and store the value to h1.
Generate the Cryptographic hash using the same certificate and using same algorithm and store the value to h2.
if both h1 and h2 hash values are same then we can certainly tell that the certificate has not been modified in transit and it actually belongs to the server what clients is talking to and client can share the "Pre Master Secret Key" for subsequent data transfer.
Below is the Mathematical representation of sample Cryptographic Hashing
m: message, c: cypher text, n: common constant , pb: public key value
pt: private key value
c = f(m, n, pb)
f’(c, n, pt) === f’(f(m, n, pb), n, pt) === m
Note: fx is a hash function on client side and f’x is a has function on server side, where fx === f’x for all data transfers
Here is the pictorial Representation of SSL handshake between Client and Server using the certificate
SClient and Server SSL Handshake
2. Important point to remember is, in all the TLS/SSL handshake Public Key playing vital role and it has to be safely placed at client end. so most of the web browsers and operating systems pre-installed with major Root Certificate authority and Intermediate Certificate Authorities certificates with public keys as shown below from chrome.
Sample Certificate details can be viewed using openssl command line (check the highlighted areas):
bash$ openssl x509 -inform der -in ca_certificate_sample.cer -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 08:3b:e0:56:90:42:46:b1:a1:75:6a:c9:59:91:c7:4a Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA Validity Not Before: Nov 10 00:00:00 2006 GMT Not After : Nov 10 00:00:00 2031 GMT Subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:e2:3b:e1:11:72:de:a8:a4:d3:a3:57:aa:50:a2: 8f:0b:77:90:c9:a2:a5:ee:12:ce:96:5b:01:09:20: cc:01:93:a7:4e:30:b7:53:f7:43:c4:69:00:57:9d: e2:8d:22:dd:87:06:40:00:81:09:ce:ce:1b:83:bf: df:cd:3b:71:46:e2:d6:66:c7:05:b3:76:27:16:8f: 7b:9e:1e:95:7d:ee:b7:48:a3:08:da:d6:af:7a:0c: 39:06:65:7f:4a:5d:1f:bc:17:f8:ab:be:ee:28:d7: 74:7f:7a:78:99:59:85:68:6e:5c:23:32:4b:bf:4e: c0:e8:5a:6d:e3:70:bf:77:10:bf:fc:01:f6:85:d9: a8:44:10:58:32:a9:75:18:d5:d1:a2:be:47:e2:27: 6a:f4:9a:33:f8:49:08:60:8b:d4:5f:b4:3a:84:bf: a1:aa:4a:4c:7d:3e:cf:4f:5f:6c:76:5e:a0:4b:37: 91:9e:dc:22:e6:6d:ce:14:1a:8e:6a:cb:fe:cd:b3: 14:64:17:c7:5b:29:9e:32:bf:f2:ee:fa:d3:0b:42: d4:ab:b7:41:32:da:0c:d4:ef:f8:81:d5:bb:8d:58: 3f:b5:1b:e8:49:28:a2:70:da:31:04:dd:f7:b2:16: f2:4c:0a:4e:07:a8:ed:4a:3d:5e:b5:7f:a3:90:c3: af:27 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Key Identifier: 03:DE:50:35:56:D1:4C:BB:66:F0:A3:E2:1B:1B:C3:97:B2:3D:D1:55 X509v3 Authority Key Identifier: keyid:03:DE:50:35:56:D1:4C:BB:66:F0:A3:E2:1B:1B:C3:97:B2:3D:D1:55
If we access any https website or url whose certificate is not found in operating system or browser CA certificates, it would show below message.
Google and most of the popular search engine and browser providers recommend the service provider to enable TLS/SSL certificate from known CA’s. if they would not get it, then they would de-prioritized in showing there information to the users.
below is the message you might see if the url is not https
What are Different Certificate Types and How to access them from programming language.
The format of the certificate we get from root CA file is base64 encoded X.509, we can use third party tools like openssh to covert X.509 to different formats. There are 2 type of encoded formats PEM encoded, DER encoded.
PEM encoding will have " — BEGIN — and — END — " for each of section like public key, private key and chain of certificates.
on the other hand DER is binary or base64 representation of PEM and mostly the extension of this would be .cer or .ctr
In general, on windows applications we might see DER encoded cer/ctr certificates used. in most of the other places PEM encoded certificate files would be used.
here is a very good explanation of different types of certificates and their usage.
On the application side we will have keystore (dont get confused with java keystore/truststore) to manage (add/remove/update) both public/private key pair and digital certificates.
There are 2 main keystores are in use
if we are using java application then we would be dealing with JKS keystore
if we are using java or other applications we would be dealing with PKCS#12(PFX). unlike JKS, we would be able to extract private keys from PKCS#12
There are two ways TLS/SSL handshake can happens.
one way handshake
In this approach server application presents a certificate contains public key with CA or self signature.
Client would check whether the server’s certificate issued is from the known trusted CA to believe that the server application is trust worthy to access and start share information.
2. Two way handshake
In this approach server and client authenticity would be checked
server presents a certificate contains public key with CA or self signature.
Client would also send a self signed or CA issues certificate in return for client’s authenticity to server.
Java way to store and access certificate information
when we are developing client or server application using java, we might come across 2 more terms keystore and truststore for managing certificates in JKS file.
keystore usually used on the server side, holds private and public key used for certificate creation and identifies the organization, and related only public key would be embedded in the certificate which would be shared with the clients.
truststore usually used on client application and holds all the known CA information.
when Server presents certificate contains CA and public key information, client would look for the CA in truststore file on his end and start the conversation if it finds an entry.
we can use "Keystore Explorer" tool or JDK provided "keytool" to view the content in the JKS.
we can also use a KeyStoreUtils java client to view and make changes to the JKS file
sample of new JKS with just certificate + public key using "Keystore Explorer"
JKS file in KeystoreExplorer
The point is if we are using java for client and server side development, we would be need import certificate files to JKS (keystore +truststore) file and set the password. for other languages we would use PKCS#12 keystore for managing the information.
below is sample code to load truststore jks file on client side (one way handshake) to check the authenticity of the server.
When the internet is ruling the world and mobile and web apps are became essential part of our life. The need of high performing, robust applications are required. which comes with redundant functionality being implemented by different applications. can we avoid rebuild same functionality which is well tested and performing well.
can we request the host application for the required information and the service serves the same information if both the party agrees to exchange the information.
RESTFull API is one of the ways to serve this kind of information. Rest API is a light weight Interface to exchange the information between client/clients and server.
REST API will have different API end points to serve each functionality. This is good in 90% of the cases.
lets discuss a requirement to query employee and department details from a service running in a remote server.
we can develop two end points one for getting the employee details ( /employee/{empid} ) and other for department details ( /department/{depid} ).
client can make first REST API call to ( /employee/{empid} ) and parse the response (may be json) to get the department id. and make the second API call to ( /department/{depid} ) by passing department id as @Pathparameters and get the department details. finally client may use just empid and name from the first call, and department id and name from the second response. all good, we got all the information required.
But here There are two potential problems with this approach
we had to make two REST API call to retrieve all the information we want, though both employee and depart entities has many-to-one relation defined at the database layer.
Though client just need empid, empname and dept id and dept name. we had to get the complete employee and department information and parse to take just what is require and leave the other data.
Lets get introduced to GraphQL:
GraphQL is a specification to enable query ability on API server to get just what is required for the client.
initially it was developed by Facebook and used as in house application. later on, this was release as a specification so that many languages like javascript, java, python and php can have its's own implementation for the specification.
the advantage of using GraphQL is we can query the related data and retrieve just the fields which client is looking for between the related models. and can avoid hitting the multiple endpoints and shipping over the complete response to client. basically it tries to address 2 basic problems discussed in the REST API.
Note: I have used Django-graphene module to explain the GraphQL. some of the code syntax would be extracted directly from the repository.
some of the main components in GraphQL
Schema:
GraphQL schema defines the implementation for Query and Mutation for API.
Register GraphQL schema:
class Query(omquery.schema.Query, graphene.ObjectType): # This class will inherit from multiple Queries # as we begin to add more apps to our project pass class Mutation(omquery.schema.Mutation, graphene.ObjectType): pass schema = graphene.Schema(query=Query, mutation=Mutation)
ObjectType:
This describers Type of the Object we return from GraphQL, every ObjectType will have a name and more importantly all the class attributes will have implicit Fields.
Object Type can be overridden by providing Meta inner-class with model object reference.
sample of GraphQL ObjectType:
class CellOMType(DjangoObjectType):
class Meta:
model = CellOM
Field:
in simplest form GraphQL is all about querying fields from the ObjectType, by default all the members of the ObjectType are mounted to Field.
each Field will have resolve_<FieldName> method attached.
sample of GraphQL Filed:
cellom = graphene.Field(CellOMType, cellid=graphene.Int()) def resolve_cellom(self, info, **kwargs): cellid = kwargs.get('cellid') if cellid is not None: return CellOM.objects.get(cellid=cellid)
List:
List is used to return multiple ObjectTypes from the resolver
Queries is the GraphQL schema to "query" different Fields available in the ObjectType (Querya fields are highlighted),
Sample code for Query schema:
class Query(graphene.ObjectType): all_celloms = graphene.List(CellOMType) def resolve_all_celloms(self, info, **kwargs): return CellOM.objects.select_related('enb').all() cellom = graphene.Field(CellOMType, cellid=graphene.Int()) def resolve_cellom(self, info, **kwargs): cellid = kwargs.get('cellid') if cellid is not None: return CellOM.objects.get(cellid=cellid)
Mutations:
Mutation is the GraphQL schema to "update/insert" Model information and persist on database.
Sample code for Mutation schema:
class EnbOMInput(graphene.InputObjectType): enbid = graphene.Int() key = graphene.String() om10 = graphene.Float() om11 = graphene.Float() om12 = graphene.Float() class CreateEnbOM(graphene.Mutation): ok = graphene.Boolean() enb = graphene.Field(EnbOMType) class Arguments: input = EnbOMInput(required=True) @staticmethod def mutate(root, info, input=None): ok = True enb_obj = EnbOM(enb=input.enbid, key=input.key, om10= input.om10, om11= input.om11, om12= input.om12) enb_obj.save() return CreateEnbOM(ok= ok, enb = enb_obj) class Mutation(graphene.ObjectType): create_enbom = CreateEnbOM.Field()
Relay
Relay is javascript framework built with GraphQL server to enable more interactive query with filters, pagination and mutation for the data.
usually Relay requires one or more "Environment" setup to fire off the requests to GraphQL to perform the necessary actions.
Graphene Django comes with Relay implementation which makes the developer life easy.
sample code snippet:
import graphene.relay as relay
from graphene_django.filter.fields import DjangoFilterConnectionField
class Query(graphene.ObjectType):
##Connect to Relay Node to apply filter filds and pagination
all_nimsrecs = DjangoFilterConnectionField(NimsTmusType)
sample Request and Response:
if we observe the request and response, it has few additional things added.
every Relay field response will have
1. unique node id: this is base64 unique id for each of the record returns from the GraphQL
2. cursor: This is used for pagination through the Relay cached results, cursor will have "first,last,before,after" to select through the result set and limit the number of results from the complete set.
3. pageinfo: This holds the cursor start and end id's to get the complete information of a single page and query the next pages
Request and Response format from GraphQL View:
Django-graphane comes with GraphQL view, which is very helpful for the GraphQL developers to test the API during development.
please refer to the code for GraphQL implementation here
