1. Overview
2. Public API (cm.downloader)
• downloaderExe.cm
• downloadApi.cm
• receiver.cm
3. Example of downloading non-catalog files using the new downloader
4. Frequently Asked Questions
• How do I force the new downloader program to close?
• How do I debug downloader problems?
• I’m getting Git errors because ConfiguraDownloader.exe is locked. How do I fix this?
• Where did all these .sha folders come from?
• How do I switch between the old and new catalogue downloader?

New Catalogue Downloader

Overview

The new Catalogue Downloader is a C# program that runs alongside CET, typically in the background, that is responsible for downloading catalogue (and other) data. It is being released in 13.5 Patch 2 (?) as an opt-in solution: that is, users will still by default use the existing catalogue downloader but can switch to trying the new one. The hope is that the new downloader will have improved stability, performance, and user-experience when compared with the existing downloader. Eventually, the new downloader will replace entirely the existing one.

Most developers should not need to do anything at all to adjust to the new downloader. However, you may want to take advantage of some of its new capabilities, namely the ability to download non-catalog data from any URI that your extension's users have access to. The interfaces for doing this are shown in the Public API section below.

Public API (cm.downloader)

downloaderExe.cm

public bool downloaderDetected()

Returns true if the downloader program is running.

downloadApi.cm

public void startDownloader(bool debug=debugModeToggle, bool runTests=false, bool verbose=false)

• debug: true if downloader should run in debug mode. This exposes some extra settings.
• runTests: if true, run a suite of tests when downloader starts
• verbose: if true, logging is more verbose

Starts the downloader program, if it isn't running already. This shouldn't generally need to be called directly, as the catalogue sync cycle and the various downloadFile functions will call it.

public void endDownloader()

Ends the downloader program, if it is running. Should almost never be used in extension code, as ongoing downloads may be lost.

public void sendDataToDownloader(CommData data, f(OutBoundMessage, str) cb=null, int tries=2, bool highPriority=false)

• data: An instance of the CommData class, encapsulating information being sent in a single ZMQ message
• f(OutboundMessage, str): Callback to run upon the downloader responding to the message
• tries: The number of times we should try to send this message before discarding it
• highPriority: If true, send this before other messages that are waiting to be sent

Send some data to the downloader program, using ZMQ. The optional callback cb will receive the response to the message, typically OK or NO.

public void downloadFile(DownloadInfo info, f(OutBoundMessage, str) cb=null)

• info: An instance of the DownloadInfo class which encapsulates the information needed to download a single file
• f(OutboundMessage, str): Callback to run upon the downloader responding to the request

Sends a request to the downloader to download one file.

public void downloadFile(str src, str tar, str grp=null, f(OutBoundMessage, str) cb=null)

• src: The source URI to download from
• tar: the target URL to download to
• grp: Optionally, the group ID that this download belongs to
• f(OutboundMessage, str): Callback to run upon the downloader responding to the request

Sends a request to the downloader to download one file.

public void downloadFiles(DownloadInfo[] infos, int limit=256, f(OutBoundMessage, str) cb=null)

• infos: A sequence of DownloadInfo instances, each encapsulating the information needed to download a single file
• limit: The maximum number of files to send in a single message. Defaults to 256
• f(OutboundMessage, str): Callback to run upon the downloader responding to the request

Sends a request to download multiple files.

receiver.cm

public void addReceiverHook(str topic, ReceiverCB hook)

• topic: The topic of this message
• RecieverCB: An alias for function(str, str), where the 1st str is the topic and the 2nd str is the message

Start listening for messages with a particular topic. The hook will receive the topic, as a string, and data, as a JSON formatted string. Some common message topics are:

• FileComplete: Any file has finished downloading
• JobComplete: Any job (batch of files) has finished downloading
• CatFileComplete: An on-demand catalog file has finished downloading
• CatJobComplete: A catalog job has finished downloading
• CatFileInstall: An on-demand catalog file requires installation by CET
• CatJobInstall: A catalog job requires installation by CET
• CatFileDuplicate: A catalog file was sent which the downloader program is already downloading

You can also subscribe to all messages by passing "*" as the topic.

public void removeReceiverHook(str topic, ReceiverCB hook)

• topic: The topic of this message
• RecieverCB: An alias for function(str, str), where the 1st str is the topic and the 2nd str is the message

Stop listening for messages with a particular topic. Must pass the same hook which was passed to addReceiverHook.

Example of downloading non-catalog files using the new downloader

use cm.downloader;
// Group downloads by an ID
str groupID = "example-group-id";
// Create a sequence of DownloadInfo
DownloadInfo[] infos();
infos << DownloadInfo("http://www.examp.le/file1.txt",
                      "C:/ExampleFolder/file1.txt",
                      grp=groupID)
      << DownloadInfo("http://www.examp.le/file2.txt",
                      "C:/ExampleFolder/file2.txt",
                      grp=groupID)
      << DownloadInfo("http://www.examp.le/file3.txt",
                      "C:/ExampleFolder/file3.txt",
                      grp=groupID);
// Send the files to the downloader. 
// This will also start the downloader if it isn't already running.
downloadFiles(infos);

Frequently Asked Questions

How do I force the new downloader program to close?

You can force the process to shutdown by finding the blue CET icon in the system tray, with the tooltip Catalogue Downloader, right-clicking said icon, and selecting Shut down. You can also find the program called ConfiguraDownloader in the Windows task manager and end the task.

How do I debug downloader problems?

If you are having issues with the new downloader, the best place to start looking is the logs. In additions to the main CET logs and the Catalogue logs, the new downloader produces its own log files. You can find these in:

C:\<Your User>\AppData\Local\ConfiguraDownloader\

The files will have the extension .log, and a name that corresponds with the date-time when that file was created. You can also access this via the downloader dialog by clicking on the button in the bottom right with the 3 dots, and then selecting Open logs folder.

If the downloader seems stuck or frozen, you can manually shut it down. See the section on force-closing the downloader. If CET is in the middle of updates, it should automatically restart the downloader when it detects it is no longer running.

I’m getting Git errors because ConfiguraDownloader.exe is locked. How do I fix this?

If you get an error like this in git, it is probably because the downloader is still running (this could still happen even if you have closed CET and emacs). In this case, you should manually close the downloader using the instructions in the section on force-closing the downloader.

Where did all these .sha folders come from?

For certain large files, the new downloader will write a cached SHA256 hash value to a file in the user’s catalog folder. These files are very small, but allow for a faster check the next time we need to determine whether a local file is out of date. The .sha folders (e.g. 643.sha) in the catalog folder contain these files.

How do I switch between the old and new catalogue downloader?

The setting to switch between the two versions of the downloader is located in the CET Control Panel, under the Catalogue tab. It is the first checkbox you will see in this tab, labeled Use new catalogue downloader.