Apache camel split xml file

Notice its the same strategy as the Aggregator supports. This Splitter can be viewed as having a build in light weight Aggregator. If you want to execute all parts in parallel you can use special notation of split with two arguments, where the second one is a boolean flag if processing should be parallel.

In Camel 2. You can split streams by enabling the streaming mode using the streaming builder method. Available as of Camel 2. Now to split this big file using XPath would cause the entire content to be loaded into memory.

So instead we can use the Tokenizer language to do this as follows:. So for example a splitted message would be as follows:. This is specified similar to the Aggregator. You can customize the underlying ThreadPoolExecutor used in the parallel splitter.

In the Java DSL try something like this:. As the Splitter can use any Expression to do the actual splitting we leverage this fact and use a method expression to invoke a Bean to get the splitted parts.

The Bean should return a value that is iterable such as: java. Collection, java. Iterator or an array. So the returned value, will then be used by Camel at runtime, to split the message. In the route we define the Expression as a method call to invoke our Bean that we have registered with the id mySplitterBean in the Registry.

And the logic for our Bean is as simple as. Notice we use Camel Bean Binding to pass in the message body as a String object. The route below illustrates this and how the split supports a aggregationStrategy to hold the in progress processed messages:. And our custom aggregationStrategy that is responsible for holding the in progress aggregated message that after the splitter is ended will be sent to the buildCombinedResponse method for final processing before the combined response can be returned to the waiting caller.

So lets run the sample and see how it works. The flow is:. The Splitter will by default continue to process the entire Exchange even in case of one of the splitted message will thrown an exception during routing. For example if you have an Exchange with rows that you split and route each sub message. During processing of these sub messages an exception is thrown at the 17th. What Camel does by default is to process the remainder messages.

You have the chance to remedy or handle this in the AggregationStrategy. But sometimes you just want Camel to stop and let the exception be propagated back, and let the Camel error handler handle it. You can do this in Camel 2. This is done by the stopOnException option as shown below:. If you omit the charset on the consumer endpoint, then Camel does not know the charset of the file, and would by default use "UTF-8".

However you can configure a JVM system property to override and use a different default encoding with the key org. In the example below this could be a problem if the files is not in UTF-8 encoding, which would be the default encoding for read the files. In this example when writing the files, the content has already been converted to a byte array, and thus would write the content directly as is without any further encodings.

You can also override and control the encoding dynamic when writing files, by setting a property on the exchange with the key Exchange. For example in the route below we set the property with a value from a message header. We suggest to keep things simpler, so if you pickup files with the same encoding, and want to write the files in a specific encoding, then favor to use the charset option on the endpoints.

Notice that if you have explicit configured a charset option on the endpoint, then that configuration is used, regardless of the Exchange. For example the route below will log the following:.

When Camel is producing files writing files there are a few gotchas affecting how to set a filename of your choice. If such a filename is not desired, then you must provide a filename in the CamelFileName message header.

The constant, Exchange. And a syntax where we set the filename on the endpoint with the fileName URI option. Filename can be set either using the expression option or as a string-based File Language expression in the CamelFileName header. See the File Language for syntax and samples.

Beware if you consume files from a folder where other applications write files to directly. Take a look at the different readLock options to see what suits your use cases. The best approach is however to write to another folder and after the write move the file in the drop folder.

You may also want to look at the doneFileName option, which uses a marker file done file to signal when a file is done and ready to be consumed. If you want only to consume files when a done file exists, then you can use the doneFileName option on the endpoint.

Will only consume files from the bar folder, if a done file exists in the same directory as the target files. However it is more common to have one done file per target file. This means there is a correlation. To do this you must use dynamic placeholders in the doneFileName option.

Currently Camel supports the following two dynamic tokens: file:name and file:name. The consumer only supports the static part of the done file name as either prefix or suffix not both. In this example only files will be polled if there exists a done file with the name file name. For example. After you have written a file you may want to write an additional done file as a kind of marker, to indicate to others that the file is finished and has been written.

To do that you can use the doneFileName option on the file producer endpoint. Will for example create a file named done-foo. Will for example create a file named foo. Listen on a directory and create a message for each file dropped there. Copy the contents to the outputdir and delete the file in the inputdir. Will scan recursively into sub-directories. Will lay out the files in the same directory structure in the outputdir as the inputdir , including any sub-directories.

If you want to store the files in the outputdir directory in the same directory, disregarding the source directory layout e. Camel will by default move any processed file into a. The body will be a File object that points to the file that was just dropped into the inputdir directory. Camel is of course also able to write files, i. In the sample below we receive some reports on the SEDA queue that we process before they are being written to a directory. Using a single route, it is possible to write a file to any number of subdirectories.

If you have a route setup as such:. You can have myBean set the header Exchange. Sometime you need to temporarily write the files to some directory relative to the destination directory.

Such situation usually happens when some external process with limited filtering capabilities is reading from the directory you are writing to. Camel supports Idempotent Consumer directly within the component so it will skip already processed files. Camel uses the absolute file name as the idempotent key, to detect duplicate files.

You can customize this key by using an expression in the idempotentKey option. For example to use both the name and the file size as the key. By default Camel uses a in memory based store for keeping track of consumed files, it uses a least recently used cache holding up to entries.

In this section we will use the file based idempotent repository org. FileIdempotentRepository instead of the in-memory based that is used as default. This repository uses a 1st level cache to avoid reading the file repository. It will only use the file repository to store the content of the 1st level cache. Thereby the repository can survive server restarts. It will load the content of the file into the 1st level cache upon startup.

The file structure is very simple as it stores the key in separate lines in the file. By default, the file store has a size limit of 1mb. When the file grows larger Camel will truncate the file store, rebuilding the content by flushing the 1st level cache into a fresh empty file. We configure our repository using Spring XML creating our file idempotent repository and define our file consumer to use our repository with the idempotentRepository using sign to indicate Registry lookup:.

In this section we will use the JPA based idempotent repository instead of the in-memory based that is used as default. MessageProcessed as model. And yes then we just need to refer to the jpaStore bean in the file consumer endpoint using the idempotentRepository using the syntax option:. Camel supports pluggable filtering strategies. You can then configure the endpoint with such a filter to skip certain files being processed. In the sample we have built our own filter that skips files starting with skip in the filename:.

And then we can configure our route using the filter attribute to reference our filter using notation that we have defined in the spring XML file:.

See the URI options above for more information. Camel supports pluggable sorting strategies. This strategy it to use the build in java. Comparator in Java. You can then configure the endpoint with such a comparator and have Camel sort the files before being processed. And then we can configure our route using the sorter option to reference to our sorter mySorter we have defined in the spring XML file:. In the Spring DSL route above notice that we can refer to beans in the Registry by prefixing the id with.

This strategy it to use the File Language to configure the sorting. The sortBy option is configured as follows:. Where each group is separated with semi colon. In the simple situations you just use one group, so a simple example could be:. This will sort by file name, you can reverse the order by prefixing reverse: to the group, so the sorting is now Z..

As we have the full power of File Language we can use some of the other parameters, so if we want to sort by file size we do:. You can configure to ignore the case, using ignoreCase: for string comparison, so if you want to use file name sorting but to ignore the case then we do:. And then we want to group by name as a 2nd option so files with same modifcation is sorted by name:. Now there is an issue here, can you spot it?

Well the modified timestamp of the file is too fine as it will be in milliseconds, but what if we want to sort by date only and then subgroup by name? Well as we have the true power of File Language we can use its date command that supports patterns. So this can be solved as:. Yeah, that is pretty powerful, oh by the way you can also use reverse per group, so we could reverse the file names:.

The option processStrategy can be used to use a custom GenericFileProcessStrategy that allows you to implement your own begin , commit and rollback logic. For instance lets assume a system writes a file in a folder you should consume. But you should not start consuming the file before another ready file has been written as well. In the begin method we can test whether the special ready file exists.

The begin method returns a boolean to indicate if we can consume the file or not. In the abort method special logic can be executed in case the begin operation returned false , for example to cleanup resources etc. The filter option allows you to implement a custom filter in Java code by implementing the org. GenericFileFilter interface. This interface has an accept method that returns a boolean. Return true to include the file, and false to skip the file. There is a isDirectory method on GenericFile whether the file is a directory.

This allows you to filter unwanted directories, to avoid traversing down unwanted directories. For example to skip any directories which starts with "skip" in the name, can be implemented as follows:. If you want to use the Camel Error Handler to deal with any exception occurring in the file consumer, then you can enable the bridgeErrorHandler option as shown below:. So all you have to do is to enable this option, and the error handler in the route will take it from there.

When using file with Spring Boot make sure to use the following Maven dependency to have support for auto configuration:. Whether to enable auto configuration of the file component. This is enabled by default. Edit this Page. File Since Camel 1. URI format file:directoryName[? Where directoryName represents the underlying file directory. Avoid reading files currently being written by another application. Configuring Options Camel components are configured on two separate levels:.

Configuring Component Options The component level is the highest level which holds general and common configurations that are inherited by the endpoints. Configuring Endpoint Options Where you find yourself configuring the most is on endpoints, as endpoints often have many options, which allows you to configure what you need the endpoint to do.

Component Options The File component supports 3 options, which are listed below. Name Description Default Type bridgeErrorHandler consumer Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler.

Query Parameters 94 parameters Name Description Default Type charset common This option is used to specify the encoding of the file. If true, the file will be deleted after it is processed successfully. If a directory, will look for files in all the sub-directories as well.

Sets the exchange pattern when the consumer creates an exchange. Sets whether synchronous processing should be strictly used. Ant style filter inclusion. Multiple inclusions may be specified in comma-delimited format. The maximum depth to traverse when recursively processing a directory. Controls if fixed delay or fixed rate is used. To shuffle the list of files sort in random order. Pluggable sorter as a java. Comparator class. Default behavior for file producer.

By default it will override any existing file, if one exist with the same name. Move and Delete operations Any move or delete operations is executed after post command the routing has completed; so during processing of the Exchange the file is still located in the inbox folder. If you want to delete the file after processing, the route should be:. Fine grained control over Move and PreMove option The move and preMove options are Expression-based, so we have the full power of the File Language to do advanced configuration of the directory and name pattern.

About moveFailed The moveFailed option allows you to move files that could not be processed successfully to another location such as an error folder of your choice. See more examples at File Language. Message Headers The following headers are supported by this component:. File producer only Header Description CamelFileName Specifies the name of the file to write relative to the endpoint directory.

CamelOverruleFileName Is used for overruling CamelFileName header and use the value instead but only once, as the producer will remove this header after writing the file. File consumer only Header Description CamelFileName Name of the consumed file as a relative file path with offset from the starting directory configured on the endpoint.

CamelFileNameOnly Only the file name the name with no leading paths. CamelFileAbsolute A boolean option specifying whether the consumed file denotes an absolute path or not. CamelFileAbsolutePath The absolute path to the file. CamelFilePath The file path.

CamelFileRelativePath The relative path. CamelFileParent The parent path. CamelFileLength A long value containing the file size. Batch Consumer This component implements the Batch Consumer. Exchange Properties, file consumer only As the file consumer implements the BatchConsumer it supports batching the files it polls. CamelBatchIndex The current index of the batch. Starts from 0. CamelBatchComplete A boolean value indicating the last Exchange in the batch.

Using charset The charset option allows for configuring an encoding of the files on both the consumer and producer endpoints. Common gotchas with folder and filenames When Camel is producing files writing files there are a few gotchas affecting how to set a filename of your choice. The sample code below produces files using the message ID as the filename:. Filename Expression Filename can be set either using the expression option or as a string-based File Language expression in the CamelFileName header.

Consuming files from folders where others drop files directly Beware if you consume files from a folder where other applications write files to directly.

Using done files Since Camel 2. Writing done files After you have written a file you may want to write an additional done file as a kind of marker, to indicate to others that the file is finished and has been written. If you want to have a control on the number of XML chunks split then you can use the group option that allows you to group the N parts together.

The token is provided in attribute token. By Ram Satish on July 31, Camel. FileInputStream; import java. InputStream; import org. CamelContext; import org. ProducerTemplate; import org.