用pig脚本调用restful服务

你看过吗 DBStorage 哪一个相似?

everything = LOAD 'categories.txt' USING PigStorage() AS (category:chararray);
...
STORE ordered INTO RestStorage('https://...');

我从来没有找到过这个问题的答案,所以我决定换个方向。我正在使用pig来加载和解析大文件,但随后将我关心的每个记录流式传输到php,以获得pig似乎不具备干净处理能力的额外处理。

它是 still not complete (read: there's a great big, very unhappy bug in the mix) 但是我认为这个概念是可靠的——只需要计算出实现细节。

everything = LOAD 'categories.txt' USING PigStorage() AS (category:chararray);
-- apply filter
-- apply filter
-- ...
-- apply last filter
ordered  = ORDER filtered_categories BY category;

streamed = STREAM limited THROUGH `php -nF process_categories.php`;
DUMP streamed;