代码之家  ›  专栏  ›  技术社区  ›  Carl Edwards monkbroc

用大型ruby/json文件填充db的最佳方法是什么?

  •  0
  • Carl Edwards monkbroc  · 技术社区  · 8 年前

    不幸的是由于文件太大 stack level too deep 每次运行任务时都会出错。我实际运行脚本的唯一方法是将文件拆分成更小的文件。尽管这样做是可行的,但是分割文件并一遍又一遍地重复任务会变得非常乏味。加载/运行大文件有什么好的替代方法吗?

    耙任务

    namespace :db do
      task populate: :environment do
        $restaurants.each_with_index do |r, index|
          uri = URI(r[:website])
    
          restaurant = Restaurant.find_or_create_by(name: r[:name], website: "#{uri.scheme}://#{uri.host}")
    
          restaurant.cuisines = r[:cuisines].map { |c| Cuisine.find_or_create_by(name: c) }
    
          location = Location.create(
            restaurant: restaurant,
            city_id: 1,
            address: r[:address],
            latitude: r[:latitude],
            longitude: r[:longitude],
            phone_number: r[:phone_number]
          )
    
          r[:hours].each do |h|
            Hour.create(
              location: location,
              day: Date::DAYNAMES.index(h[:day]),
              opens: h[:opens],
              closes: h[:closes]
            )
          end
    
          menu_group = MenuGroup.create(
            restaurant: restaurant,
            locations: [location],
            address: r[:address]
          )
    
          r[:menus].each do |m|
            menu = Menu.create(
              menu_group: menu_group,
              position: m[:position],
              name: m[:name]
            )
    
            m[:sections].each do |s|
              section = Section.create(
                menu: menu,
                position: s[:position],
                name: s[:name]
              )
    
              s[:dishes].each do |d|
                tag = Tag.find_or_create_by(
                  name: d[:name].downcase.strip
                )
    
                Dish.find_or_create_by(
                  restaurant: restaurant,
                  sections: [section],
                  tags: [tag],
                  name: d[:name],
                  description: d[:description]
                )
              end
            end
          end
    
          puts "#{index + 1} of #{$restaurants.size} completed"
        end
      end
    end
    

    错误

    rake aborted!
    SystemStackError: stack level too deep
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:12:in`to_binary'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:12:in`input_to_storage'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:37:in`fetch'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:37:in`load_iseq'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
    /usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:283:in `block in require'
    /usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:249:in `load_dependency'
    /usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:283:in `require'
    /Users/user/app/lib/tasks/populate.rake:1:in `<main>'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
    /usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
    /usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:277:in `block in load'
    /usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:249:in `load_dependency'
    /usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:277:in `load'
    /usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:650:in `block in run_tasks_blocks'
    /usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:650:in `each'
    /usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:650:in `run_tasks_blocks'
    /usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/application.rb:515:in `run_tasks_blocks'
    /usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:459:in `load_tasks'
    /Users/user/app/Rakefile:6:in `<top (required)>'
    /usr/local/lib/ruby/gems/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
    (See full trace by running task with --trace)
    
    1 回复  |  直到 8 年前
        1
  •  0
  •   NiftyAsp    8 年前

    我会用这样的东西 Sidekiq 把工作分配给能同时工作的工人。

    例如:

    $restaurants.each_with_index do |r, index|
        RestaurantParser.perform_async(r, index)
    end
    

    在RestaurantParser中,执行通常要执行的步骤。

    只要餐厅不依赖DB中已有的其他餐厅,您就可以同时运行工人来加快流程。

    推荐文章