[{"data":1,"prerenderedAt":1403},["ShallowReactive",2],{"blog-current-how-to-use-different-dns-servers-on-specific-domains-en":3,"blog-previous-how-to-use-different-dns-servers-on-specific-domains-en":602,"blog-next-how-to-use-different-dns-servers-on-specific-domains-en":616,"blog-alt-de-how-to-use-different-dns-servers-on-specific-domains-en":632,"blog-alt-en-how-to-use-different-dns-servers-on-specific-domains-en":634,"employee-bernd-helm":635,"content-query-ViehLtZIqO":741,"content-query-Dmf1CQj4Id":1270,"content-query-1PFeYVQSzn":1341,"related-refs-devops--en":1377},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"author":10,"image":11,"releaseDate":12,"blogCategories":13,"articleTags":16,"tags":18,"body":20,"_type":596,"_id":597,"_source":598,"_file":599,"_stem":600,"_extension":601},"/en/blog/how-to-use-different-dns-servers-on-specific-domains","blog",false,"","How to use different DNS servers for specific domains name resolution – VPN Use-Case","How to resolve selected domains over VPN on linux. In today's world, more people than ever use VPN services to work from remote. However, in some cases it's not desirable to route all traffic and all domain name resolutions over the VPN connection.","bernd-helm","/blog/thumbnails/VPN.png","2022-04-21",[14,15],"What moves us","DevOps",[17],"VPN",[19],"devops",{"type":21,"children":22,"toc":591},"root",[23,32,38,43,65,78,84,113,130,136,141,154,167,200,235,256,313,330,357,400,429,479,498,511,553,565,585],{"type":24,"tag":25,"props":26,"children":28},"element","h2",{"id":27},"how-to-resolve-selected-domains-over-vpn-on-linux",[29],{"type":30,"value":31},"text","How to resolve selected domains over VPN on Linux",{"type":24,"tag":33,"props":34,"children":35},"p",{},[36],{"type":30,"value":37},"In today's world, more people than ever use VPN services to work from remote. However, in some cases it's not desirable\nto route all traffic and all domain name resolutions over the VPN connection. Even if the VPN-Server wants the client to\nconfigure itself to work like this, the client can be configured to ignore the request to route all traffic over the VPN\nconnection.",{"type":24,"tag":33,"props":39,"children":40},{},[41],{"type":30,"value":42},"For example, with Openvpn the option",{"type":24,"tag":44,"props":45,"children":49},"pre",{"className":46,"code":47,"language":48,"meta":7,"style":7},"language-apache shiki shiki-themes github-dark github-dark monokai","pull-filter ignore redirect-gateway\n","apache",[50],{"type":24,"tag":51,"props":52,"children":53},"code",{"__ignoreMap":7},[54],{"type":24,"tag":55,"props":56,"children":59},"span",{"class":57,"line":58},"line",1,[60],{"type":24,"tag":55,"props":61,"children":63},{"style":62},"--shiki-default:#E1E4E8;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2",[64],{"type":30,"value":47},{"type":24,"tag":33,"props":66,"children":67},{},[68,70,76],{"type":30,"value":69},"can be used to tell the ",{"type":24,"tag":51,"props":71,"children":73},{"className":72},[],[74],{"type":30,"value":75},"openvpn-client",{"type":30,"value":77}," to ignore all \"route all\" requests from the server.",{"type":24,"tag":25,"props":79,"children":81},{"id":80},"fritzbox-example",[82],{"type":30,"value":83},"FRITZ!Box example",{"type":24,"tag":33,"props":85,"children":86},{},[87,89,95,97,103,105,111],{"type":30,"value":88},"Recently I had to use a VPN-Connection to a ",{"type":24,"tag":90,"props":91,"children":92},"strong",{},[93],{"type":30,"value":94},"FRITZ!Box",{"type":30,"value":96},". It's a very popular router in Germany that not only offers\neasy\nVPN-Connections but also adds all hosts on its network to its own dns. If you want to talk to a machine with the name ",{"type":24,"tag":51,"props":98,"children":100},{"className":99},[],[101],{"type":30,"value":102}," workstation",{"type":30,"value":104}," you can reach it using the \"workstation.fritz.box\" dns name. The fritz-box itself is also available on the\ndns name ",{"type":24,"tag":51,"props":106,"children":108},{"className":107},[],[109],{"type":30,"value":110},"fritz.box",{"type":30,"value":112},".",{"type":24,"tag":33,"props":114,"children":115},{},[116,118,123,125,129],{"type":30,"value":117},"In my case, I prefer to use my own DNS server for all queries because of speed and privacy reasons - that means, I only\nwant the\ndomains with the ",{"type":24,"tag":51,"props":119,"children":121},{"className":120},[],[122],{"type":30,"value":110},{"type":30,"value":124}," suffix to resolve over the ",{"type":24,"tag":90,"props":126,"children":127},{},[128],{"type":30,"value":94},{"type":30,"value":112},{"type":24,"tag":25,"props":131,"children":133},{"id":132},"solution-using-a-local-dns-server-dnsmasq",[134],{"type":30,"value":135},"Solution using a local DNS server - dnsmasq",{"type":24,"tag":33,"props":137,"children":138},{},[139],{"type":30,"value":140},"Dnsmasq is a lightweight DNS-Server that you can run on your machine to get control on how the name resolution works. A\nnice side effect is that it has its own dns cache, making recurring dns queries faster. Here is how I have set it up:",{"type":24,"tag":33,"props":142,"children":143},{},[144,146,152],{"type":30,"value":145},"Before you start you need to find out which DNS server is used on the VPN. It's usually the default gateway that usually\nhas a ",{"type":24,"tag":51,"props":147,"children":149},{"className":148},[],[150],{"type":30,"value":151},".1",{"type":30,"value":153}," at the end. You can watch the VPN logs closely to learn which DNS server gets pushed on connect.",{"type":24,"tag":33,"props":155,"children":156},{},[157,159,165],{"type":30,"value":158},"There is a tool called ",{"type":24,"tag":51,"props":160,"children":162},{"className":161},[],[163],{"type":30,"value":164},"dig",{"type":30,"value":166}," which can do dns queries over specified dns servers, for example",{"type":24,"tag":44,"props":168,"children":172},{"className":169,"code":170,"language":171,"meta":7,"style":7},"language-bash shiki shiki-themes github-dark github-dark monokai","dig @192.168.1.1 a fritz.box\n","bash",[173],{"type":24,"tag":51,"props":174,"children":175},{"__ignoreMap":7},[176],{"type":24,"tag":55,"props":177,"children":178},{"class":57,"line":58},[179,184,190,195],{"type":24,"tag":55,"props":180,"children":182},{"style":181},"--shiki-default:#B392F0;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E",[183],{"type":30,"value":164},{"type":24,"tag":55,"props":185,"children":187},{"style":186},"--shiki-default:#9ECBFF;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74",[188],{"type":30,"value":189}," @192.168.1.1",{"type":24,"tag":55,"props":191,"children":192},{"style":186},[193],{"type":30,"value":194}," a",{"type":24,"tag":55,"props":196,"children":197},{"style":186},[198],{"type":30,"value":199}," fritz.box\n",{"type":24,"tag":33,"props":201,"children":202},{},[203,205,211,213,218,220,226,228,234],{"type":30,"value":204},"will ask the dns server ",{"type":24,"tag":51,"props":206,"children":208},{"className":207},[],[209],{"type":30,"value":210},"192.168.1.1",{"type":30,"value":212}," for the ip of ",{"type":24,"tag":51,"props":214,"children":216},{"className":215},[],[217],{"type":30,"value":110},{"type":30,"value":219},". ",{"type":24,"tag":51,"props":221,"children":223},{"className":222},[],[224],{"type":30,"value":225},"Dig",{"type":30,"value":227}," is very handy when debugging dns problems and to\ntest\nyour setup. You may have to install it on your System. On debian, it lives in a package called ",{"type":24,"tag":51,"props":229,"children":231},{"className":230},[],[232],{"type":30,"value":233},"dnsutils",{"type":30,"value":112},{"type":24,"tag":33,"props":236,"children":237},{},[238,240,246,248,254],{"type":30,"value":239},"First install ",{"type":24,"tag":51,"props":241,"children":243},{"className":242},[],[244],{"type":30,"value":245},"dnsmasq",{"type":30,"value":247}," on your Linux and edit its configuration (usually lives in ",{"type":24,"tag":51,"props":249,"children":251},{"className":250},[],[252],{"type":30,"value":253},"/etc/dnsmasq.conf",{"type":30,"value":255},"). Add these lines\nhere:",{"type":24,"tag":44,"props":257,"children":259},{"className":46,"code":258,"language":48,"meta":7,"style":7},"resolv-file=/etc/resolv.dnsmasq.conf\nserver=/fritz.box/192.168.1.1\n",[260],{"type":24,"tag":51,"props":261,"children":262},{"__ignoreMap":7},[263,271],{"type":24,"tag":55,"props":264,"children":265},{"class":57,"line":58},[266],{"type":24,"tag":55,"props":267,"children":268},{"style":62},[269],{"type":30,"value":270},"resolv-file=/etc/resolv.dnsmasq.conf\n",{"type":24,"tag":55,"props":272,"children":274},{"class":57,"line":273},2,[275,280,286,290,295,299,304,308],{"type":24,"tag":55,"props":276,"children":277},{"style":62},[278],{"type":30,"value":279},"server=/fritz.box/",{"type":24,"tag":55,"props":281,"children":283},{"style":282},"--shiki-default:#79B8FF;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF",[284],{"type":30,"value":285},"192",{"type":24,"tag":55,"props":287,"children":288},{"style":62},[289],{"type":30,"value":112},{"type":24,"tag":55,"props":291,"children":292},{"style":282},[293],{"type":30,"value":294},"168",{"type":24,"tag":55,"props":296,"children":297},{"style":62},[298],{"type":30,"value":112},{"type":24,"tag":55,"props":300,"children":301},{"style":282},[302],{"type":30,"value":303},"1",{"type":24,"tag":55,"props":305,"children":306},{"style":62},[307],{"type":30,"value":112},{"type":24,"tag":55,"props":309,"children":310},{"style":282},[311],{"type":30,"value":312},"1\n",{"type":24,"tag":33,"props":314,"children":315},{},[316,321,323,328],{"type":24,"tag":51,"props":317,"children":319},{"className":318},[],[320],{"type":30,"value":110},{"type":30,"value":322}," is the domain you want to resolve over the VPN-Dns-server, ",{"type":24,"tag":51,"props":324,"children":326},{"className":325},[],[327],{"type":30,"value":210},{"type":30,"value":329}," in this example.",{"type":24,"tag":33,"props":331,"children":332},{},[333,335,340,342,347,349,355],{"type":30,"value":334},"The ",{"type":24,"tag":90,"props":336,"children":337},{},[338],{"type":30,"value":339},"Resolv-File",{"type":30,"value":341}," will tell ",{"type":24,"tag":51,"props":343,"children":345},{"className":344},[],[346],{"type":30,"value":245},{"type":30,"value":348}," how it should resolve its dns queries in case there are no other rules, so let's\ncreate\na ",{"type":24,"tag":51,"props":350,"children":352},{"className":351},[],[353],{"type":30,"value":354},"/etc/resolv.dnsmasq.conf",{"type":30,"value":356},"\nwith content",{"type":24,"tag":44,"props":358,"children":360},{"className":46,"code":359,"language":48,"meta":7,"style":7},"nameserver 1.1.1.1\n",[361],{"type":24,"tag":51,"props":362,"children":363},{"__ignoreMap":7},[364],{"type":24,"tag":55,"props":365,"children":366},{"class":57,"line":58},[367,372,376,380,384,388,392,396],{"type":24,"tag":55,"props":368,"children":369},{"style":62},[370],{"type":30,"value":371},"nameserver ",{"type":24,"tag":55,"props":373,"children":374},{"style":282},[375],{"type":30,"value":303},{"type":24,"tag":55,"props":377,"children":378},{"style":62},[379],{"type":30,"value":112},{"type":24,"tag":55,"props":381,"children":382},{"style":282},[383],{"type":30,"value":303},{"type":24,"tag":55,"props":385,"children":386},{"style":62},[387],{"type":30,"value":112},{"type":24,"tag":55,"props":389,"children":390},{"style":282},[391],{"type":30,"value":303},{"type":24,"tag":55,"props":393,"children":394},{"style":62},[395],{"type":30,"value":112},{"type":24,"tag":55,"props":397,"children":398},{"style":282},[399],{"type":30,"value":312},{"type":24,"tag":33,"props":401,"children":402},{},[403,405,411,413,419,421,427],{"type":30,"value":404},"If you use a static network configuration, you now can just edit it to use ",{"type":24,"tag":51,"props":406,"children":408},{"className":407},[],[409],{"type":30,"value":410},"127.0.0.1",{"type":30,"value":412}," as its nameserver. If you use\nautomatic ip configuration using dhcp, you need to configure your dhcp client use the new local dns server. On debian,\nthe default is ",{"type":24,"tag":51,"props":414,"children":416},{"className":415},[],[417],{"type":30,"value":418},"dhclient",{"type":30,"value":420},". In that case edit ",{"type":24,"tag":51,"props":422,"children":424},{"className":423},[],[425],{"type":30,"value":426},"/etc/dhcp/dhclient.conf",{"type":30,"value":428}," and add:",{"type":24,"tag":44,"props":430,"children":432},{"className":46,"code":431,"language":48,"meta":7,"style":7},"prepend domain-name-servers 127.0.0.1;\n",[433],{"type":24,"tag":51,"props":434,"children":435},{"__ignoreMap":7},[436],{"type":24,"tag":55,"props":437,"children":438},{"class":57,"line":58},[439,444,449,453,458,462,466,470,474],{"type":24,"tag":55,"props":440,"children":441},{"style":62},[442],{"type":30,"value":443},"prepend domain-name-servers ",{"type":24,"tag":55,"props":445,"children":446},{"style":282},[447],{"type":30,"value":448},"127",{"type":24,"tag":55,"props":450,"children":451},{"style":62},[452],{"type":30,"value":112},{"type":24,"tag":55,"props":454,"children":455},{"style":282},[456],{"type":30,"value":457},"0",{"type":24,"tag":55,"props":459,"children":460},{"style":62},[461],{"type":30,"value":112},{"type":24,"tag":55,"props":463,"children":464},{"style":282},[465],{"type":30,"value":457},{"type":24,"tag":55,"props":467,"children":468},{"style":62},[469],{"type":30,"value":112},{"type":24,"tag":55,"props":471,"children":472},{"style":282},[473],{"type":30,"value":303},{"type":24,"tag":55,"props":475,"children":476},{"style":62},[477],{"type":30,"value":478},";\n",{"type":24,"tag":33,"props":480,"children":481},{},[482,484,489,491,496],{"type":30,"value":483},"That's it. All you have to do now is to restart ",{"type":24,"tag":51,"props":485,"children":487},{"className":486},[],[488],{"type":30,"value":245},{"type":30,"value":490}," so it reloads its config and reconnect/restart ",{"type":24,"tag":51,"props":492,"children":494},{"className":493},[],[495],{"type":30,"value":418},{"type":30,"value":497},"\n(disconnecting and reconnecting the network may work, reboot the machine when in doubt).",{"type":24,"tag":33,"props":499,"children":500},{},[501,503,509],{"type":30,"value":502},"Now you can check if the file ",{"type":24,"tag":51,"props":504,"children":506},{"className":505},[],[507],{"type":30,"value":508},"/etc/resolv.conf",{"type":30,"value":510}," has",{"type":24,"tag":44,"props":512,"children":514},{"className":46,"code":513,"language":48,"meta":7,"style":7},"nameserver 127.0.0.1\n",[515],{"type":24,"tag":51,"props":516,"children":517},{"__ignoreMap":7},[518],{"type":24,"tag":55,"props":519,"children":520},{"class":57,"line":58},[521,525,529,533,537,541,545,549],{"type":24,"tag":55,"props":522,"children":523},{"style":62},[524],{"type":30,"value":371},{"type":24,"tag":55,"props":526,"children":527},{"style":282},[528],{"type":30,"value":448},{"type":24,"tag":55,"props":530,"children":531},{"style":62},[532],{"type":30,"value":112},{"type":24,"tag":55,"props":534,"children":535},{"style":282},[536],{"type":30,"value":457},{"type":24,"tag":55,"props":538,"children":539},{"style":62},[540],{"type":30,"value":112},{"type":24,"tag":55,"props":542,"children":543},{"style":282},[544],{"type":30,"value":457},{"type":24,"tag":55,"props":546,"children":547},{"style":62},[548],{"type":30,"value":112},{"type":24,"tag":55,"props":550,"children":551},{"style":282},[552],{"type":30,"value":312},{"type":24,"tag":33,"props":554,"children":555},{},[556,558,563],{"type":30,"value":557},"as first nameserver. If it has, the selected domain (",{"type":24,"tag":51,"props":559,"children":561},{"className":560},[],[562],{"type":30,"value":110},{"type":30,"value":564}," in my example) should now resolve.",{"type":24,"tag":33,"props":566,"children":567},{},[568,570,575,577,583],{"type":30,"value":569},"On ",{"type":24,"tag":51,"props":571,"children":573},{"className":572},[],[574],{"type":30,"value":245},{"type":30,"value":576},", you can add multiple ",{"type":24,"tag":51,"props":578,"children":580},{"className":579},[],[581],{"type":30,"value":582},"server=",{"type":30,"value":584}," lines, which comes in handy if you have multiple domains that need to be\nresolved over different DNS servers.",{"type":24,"tag":586,"props":587,"children":588},"style",{},[589],{"type":30,"value":590},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":7,"searchDepth":273,"depth":273,"links":592},[593,594,595],{"id":27,"depth":273,"text":31},{"id":80,"depth":273,"text":83},{"id":132,"depth":273,"text":135},"markdown","common:en:blog:10.how-to-use-different-dns-servers-on-specific-domains.md","common","en/blog/10.how-to-use-different-dns-servers-on-specific-domains.md","en/blog/10.how-to-use-different-dns-servers-on-specific-domains","md",{"_path":603,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":604,"description":605,"author":10,"image":606,"releaseDate":607,"blogCategories":608,"articleTags":610,"tags":611,"_type":596,"_id":613,"_source":598,"_file":614,"_stem":615,"_extension":601},"/en/blog/shopware-5-parallel-thumbnail-generation","Shopware 5 – High-Speed Parallel Thumbnail Generation","How to tune the integrated SW5 mechanism to work more then 22 times faster! Shopware 5 - parallel thumbnail generation after moving a Shopware 5 system to another server. We had a customer with 400k images and 1600k thumbnails that needed to move from an old HDD based server to a new SSD …","/images/shopware.svg","2021-04-29",[609,15],"Shopware",[17],[612],"shopware","common:en:blog:9.shopware-5-parallel-thumbnail-generation.md","en/blog/9.shopware-5-parallel-thumbnail-generation.md","en/blog/9.shopware-5-parallel-thumbnail-generation",{"_path":617,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":618,"description":619,"author":10,"image":620,"releaseDate":621,"blogCategories":622,"articleTags":623,"tags":626,"_type":596,"_id":629,"_source":598,"_file":630,"_stem":631,"_extension":601},"/en/blog/amazon-mws-sp-api-wrapper","Amazon MWS to SP API Wrapper","We have released a wrapper that converts Amazon MWS to SP API, offering developers with outdated systems a simple solution. This open-source project provides a practical alternative to redevelopment and supports the continuation of existing applications.","/blog/thumbnails/AWS.svg","2024-02-21",[14,15],[624,625],"Amazon","Open Source",[627,628],"e-commerce","backend","common:en:blog:11.amazon-mws-sp-api-wrapper.md","en/blog/11.amazon-mws-sp-api-wrapper.md","en/blog/11.amazon-mws-sp-api-wrapper",{"_path":633},"/blog/wie-man-verschiedene-dns-server-fuer-bestimter-domaens-verwendet",{"_path":4},{"_path":636,"_dir":637,"_draft":6,"_partial":6,"_locale":7,"slug":10,"teams":638,"primaryTeam":640,"firstName":641,"lastName":642,"prefixTitle":7,"suffixTitle":7,"education":643,"executiveRole":648,"role":649,"workingSince":657,"inTheCompanySince":658,"techSkills":659,"skills":705,"projects":718,"contactDetails":731,"_image":734,"image":735,"_id":736,"_type":737,"title":738,"_source":637,"_file":739,"_stem":740,"_extension":737},"/employees/bernd-helm","employees",[639,640],"ai","devOps","Bernd","Helm",[644],[645,646,647],"B. Sc. Angewandte Informatik","FHDW Dresden","2010","CTO",[650,651,640,652,653,654,655,656],"founder","chiefTechnologyOfficer","databaseSpecialist","admin","softwareDeveloper","backendDeveloper","consultant","2005","2008",[660,664,667,669,672,674,677,679,681,683,686,690,693,696,699,702],{"name":661,"level":662,"icon":663},"Docker","expert","/images/Docker.svg",{"name":665,"level":662,"icon":666},"Linux","/images/linux_os-mono.svg",{"name":668,"level":662},"Zabbix",{"name":670,"level":662,"icon":671},"MariaDB ColumnStore","/images/maria-db-logo.svg",{"name":673,"level":662},"OpenAI",{"name":675,"level":662,"icon":676},"Pytorch","/images/PyTorch.svg",{"name":678,"level":662},"PHP",{"name":680,"level":662},"Java",{"name":682,"level":662},"Python",{"name":684,"level":662,"icon":685},"SQL","/images/SQL.svg",{"name":687,"level":688,"icon":689},"C++","advanced","/images/cpp-logo.svg",{"name":691,"level":688,"icon":692},"C#","/images/csharp.svg",{"name":694,"level":688,"icon":695},"CSS","/images/css.svg",{"name":697,"level":688,"icon":698},"HTML","/images/html.svg",{"name":700,"level":688,"icon":701},"OpenCV","/images/OpenCV.svg",{"name":703,"level":688,"icon":704},"Vue.js","/images/vuejs.svg",[706,708,710,712,714,716],{"name":707,"level":662},"artificialIntelligence",{"name":709,"level":662},"codingGuidelines",{"name":711,"level":662},"databases",{"name":713,"level":662},"linuxServerAdministration",{"name":715,"level":662},"softwareArchitect",{"name":717,"level":688},"qualityAssurance",[719,722,724,726,729],{"project":720,"position":721},"Gridside","Technical Consultant",{"project":723,"position":721},"Herole",{"project":725,"position":721},"Montagespezis",{"project":727,"position":728},"Orsee","Technical Manager",{"project":730,"position":721},"Vipr",{"eMail":732,"phone":733,"visibility":303},"bernd.helm@helmundwalter.de","+49 351 799 035 20","images/employees/Portraits/bernd_helm.webp","images/employees/Portraits/BerndHelm_MS.webp","employees:employees:1.bernd-helm.json","json","Bernd Helm","employees/1.bernd-helm.json","employees/1.bernd-helm",{"_path":603,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":604,"description":605,"author":10,"image":606,"releaseDate":607,"blogCategories":742,"articleTags":743,"tags":744,"body":745,"_type":596,"_id":613,"_source":598,"_file":614,"_stem":615,"_extension":601},[609,15],[17],[612],{"type":21,"children":746,"toc":1265},[747,753,758,770,775,781,786,806,815,820,825,1165,1170,1175,1180,1185,1197,1247,1255,1261],{"type":24,"tag":25,"props":748,"children":750},{"id":749},"shopware-5-parallel-thumbnail-generation-after-moving-a-shopware-5-system-to-another-server",[751],{"type":30,"value":752},"Shopware 5 - parallel thumbnail generation after moving a Shopware 5 system to another server",{"type":24,"tag":33,"props":754,"children":755},{},[756],{"type":30,"value":757},"We had a customer with 400k images and 1600k thumbnails that needed to move from an old hdd based server to a new ssd based one. The problem was that the old server was so slow that it already needed two days to count through all images, not speaking about coping them.",{"type":24,"tag":33,"props":759,"children":760},{},[761,763],{"type":30,"value":762},"So we decided to copy only the original images and regenerate the Thumbnails. For copying the original images, I created a small console command that exports all paths of the original images that we need to copy: ",{"type":24,"tag":764,"props":765,"children":767},"a",{"href":766},"https://gist.github.com/bhelm/2d30f0cebcf4a7d8ea41c532ec67cd62",[768],{"type":30,"value":769},"ExportImagesCommand.php",{"type":24,"tag":33,"props":771,"children":772},{},[773],{"type":30,"value":774},"This filelist can be used with tar -T or rsync --files-from= options that tells these tools to only process the listed files. for the initial copy process, tar is highly recommended, as it just picks up the files listed without doing any \"calculation\" as rsync does.",{"type":24,"tag":25,"props":776,"children":778},{"id":777},"sw5-default-thumbnail-generation-would-have-taken-80-hours",[779],{"type":30,"value":780},"SW5 default thumbnail generation would have taken 80 hours",{"type":24,"tag":33,"props":782,"children":783},{},[784],{"type":30,"value":785},"... and would only use half of a core.",{"type":24,"tag":33,"props":787,"children":788},{},[789,791,797,799,804],{"type":30,"value":790},"I was curious if I can speed up this generation process. The server itself has 32 cores available, so I copied the ",{"type":24,"tag":792,"props":793,"children":794},"em",{},[795],{"type":30,"value":796},"generate thumbnail",{"type":30,"value":798}," command from sw5 and modified it to work in batches with an ",{"type":24,"tag":792,"props":800,"children":801},{},[802],{"type":30,"value":803},"--batch",{"type":30,"value":805}," parameter:",{"type":24,"tag":33,"props":807,"children":808},{},[809],{"type":24,"tag":764,"props":810,"children":812},{"href":811},"https://gist.github.com/bhelm/2015d3829d4a3f24f9760f6e4e1aac1f",[813],{"type":30,"value":814},"ParallelThumbnailGenerateCommand.php",{"type":24,"tag":33,"props":816,"children":817},{},[818],{"type":30,"value":819},"To make it work, I just modified the Shopware core at engine/Shopware/Models/Media/Repository.php",{"type":24,"tag":33,"props":821,"children":822},{},[823],{"type":30,"value":824},"I just changed the getAlbumMediaQuery function to:",{"type":24,"tag":44,"props":826,"children":830},{"className":827,"code":828,"language":829,"meta":7,"style":7},"language-cpp shiki shiki-themes github-dark github-dark monokai","public function getAlbumMediaQuery($albumId, $filter = null, $orderBy = null, $offset = null, $limit = null, $validTypes = null, $batch = null)\n{\n$builder = $this->getAlbumMediaQueryBuilder($albumId, $filter, $orderBy, $validTypes);\nif (is_numeric($batch)) {\n$builder->andWhere('MOD(media.id, 1000) = ?3');\n$builder->setParameter(3, $batch);\n}\n\n    if ($limit !== null) {\n        $builder->setFirstResult($offset)\n                ->setMaxResults($limit);\n    }\n\n    return $builder->getQuery();\n}\n","cpp",[831],{"type":24,"tag":51,"props":832,"children":833},{"__ignoreMap":7},[834,908,916,955,979,1008,1035,1044,1054,1078,1097,1116,1125,1133,1157],{"type":24,"tag":55,"props":835,"children":836},{"class":57,"line":58},[837,842,847,852,858,863,867,872,876,881,885,890,894,899,903],{"type":24,"tag":55,"props":838,"children":839},{"style":62},[840],{"type":30,"value":841},"public function ",{"type":24,"tag":55,"props":843,"children":844},{"style":181},[845],{"type":30,"value":846},"getAlbumMediaQuery",{"type":24,"tag":55,"props":848,"children":849},{"style":62},[850],{"type":30,"value":851},"($albumId, $filter ",{"type":24,"tag":55,"props":853,"children":855},{"style":854},"--shiki-default:#F97583;--shiki-dark:#F97583;--shiki-sepia:#F92672",[856],{"type":30,"value":857},"=",{"type":24,"tag":55,"props":859,"children":860},{"style":62},[861],{"type":30,"value":862}," null, $orderBy ",{"type":24,"tag":55,"props":864,"children":865},{"style":854},[866],{"type":30,"value":857},{"type":24,"tag":55,"props":868,"children":869},{"style":62},[870],{"type":30,"value":871}," null, $offset ",{"type":24,"tag":55,"props":873,"children":874},{"style":854},[875],{"type":30,"value":857},{"type":24,"tag":55,"props":877,"children":878},{"style":62},[879],{"type":30,"value":880}," null, $limit ",{"type":24,"tag":55,"props":882,"children":883},{"style":854},[884],{"type":30,"value":857},{"type":24,"tag":55,"props":886,"children":887},{"style":62},[888],{"type":30,"value":889}," null, $validTypes ",{"type":24,"tag":55,"props":891,"children":892},{"style":854},[893],{"type":30,"value":857},{"type":24,"tag":55,"props":895,"children":896},{"style":62},[897],{"type":30,"value":898}," null, $batch ",{"type":24,"tag":55,"props":900,"children":901},{"style":854},[902],{"type":30,"value":857},{"type":24,"tag":55,"props":904,"children":905},{"style":62},[906],{"type":30,"value":907}," null)\n",{"type":24,"tag":55,"props":909,"children":910},{"class":57,"line":273},[911],{"type":24,"tag":55,"props":912,"children":913},{"style":62},[914],{"type":30,"value":915},"{\n",{"type":24,"tag":55,"props":917,"children":919},{"class":57,"line":918},3,[920,925,929,934,940,945,950],{"type":24,"tag":55,"props":921,"children":922},{"style":62},[923],{"type":30,"value":924},"$builder ",{"type":24,"tag":55,"props":926,"children":927},{"style":854},[928],{"type":30,"value":857},{"type":24,"tag":55,"props":930,"children":931},{"style":62},[932],{"type":30,"value":933}," $",{"type":24,"tag":55,"props":935,"children":937},{"style":936},"--shiki-default:#79B8FF;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F",[938],{"type":30,"value":939},"this",{"type":24,"tag":55,"props":941,"children":942},{"style":62},[943],{"type":30,"value":944},"->",{"type":24,"tag":55,"props":946,"children":947},{"style":181},[948],{"type":30,"value":949},"getAlbumMediaQueryBuilder",{"type":24,"tag":55,"props":951,"children":952},{"style":62},[953],{"type":30,"value":954},"($albumId, $filter, $orderBy, $validTypes);\n",{"type":24,"tag":55,"props":956,"children":958},{"class":57,"line":957},4,[959,964,969,974],{"type":24,"tag":55,"props":960,"children":961},{"style":854},[962],{"type":30,"value":963},"if",{"type":24,"tag":55,"props":965,"children":966},{"style":62},[967],{"type":30,"value":968}," (",{"type":24,"tag":55,"props":970,"children":971},{"style":181},[972],{"type":30,"value":973},"is_numeric",{"type":24,"tag":55,"props":975,"children":976},{"style":62},[977],{"type":30,"value":978},"($batch)) {\n",{"type":24,"tag":55,"props":980,"children":982},{"class":57,"line":981},5,[983,988,993,998,1003],{"type":24,"tag":55,"props":984,"children":985},{"style":62},[986],{"type":30,"value":987},"$builder->",{"type":24,"tag":55,"props":989,"children":990},{"style":181},[991],{"type":30,"value":992},"andWhere",{"type":24,"tag":55,"props":994,"children":995},{"style":62},[996],{"type":30,"value":997},"(",{"type":24,"tag":55,"props":999,"children":1000},{"style":186},[1001],{"type":30,"value":1002},"'MOD(media.id, 1000) = ?3'",{"type":24,"tag":55,"props":1004,"children":1005},{"style":62},[1006],{"type":30,"value":1007},");\n",{"type":24,"tag":55,"props":1009,"children":1011},{"class":57,"line":1010},6,[1012,1016,1021,1025,1030],{"type":24,"tag":55,"props":1013,"children":1014},{"style":62},[1015],{"type":30,"value":987},{"type":24,"tag":55,"props":1017,"children":1018},{"style":181},[1019],{"type":30,"value":1020},"setParameter",{"type":24,"tag":55,"props":1022,"children":1023},{"style":62},[1024],{"type":30,"value":997},{"type":24,"tag":55,"props":1026,"children":1027},{"style":282},[1028],{"type":30,"value":1029},"3",{"type":24,"tag":55,"props":1031,"children":1032},{"style":62},[1033],{"type":30,"value":1034},", $batch);\n",{"type":24,"tag":55,"props":1036,"children":1038},{"class":57,"line":1037},7,[1039],{"type":24,"tag":55,"props":1040,"children":1041},{"style":62},[1042],{"type":30,"value":1043},"}\n",{"type":24,"tag":55,"props":1045,"children":1047},{"class":57,"line":1046},8,[1048],{"type":24,"tag":55,"props":1049,"children":1051},{"emptyLinePlaceholder":1050},true,[1052],{"type":30,"value":1053},"\n",{"type":24,"tag":55,"props":1055,"children":1057},{"class":57,"line":1056},9,[1058,1063,1068,1073],{"type":24,"tag":55,"props":1059,"children":1060},{"style":854},[1061],{"type":30,"value":1062},"    if",{"type":24,"tag":55,"props":1064,"children":1065},{"style":62},[1066],{"type":30,"value":1067}," ($limit ",{"type":24,"tag":55,"props":1069,"children":1070},{"style":854},[1071],{"type":30,"value":1072},"!==",{"type":24,"tag":55,"props":1074,"children":1075},{"style":62},[1076],{"type":30,"value":1077}," null) {\n",{"type":24,"tag":55,"props":1079,"children":1081},{"class":57,"line":1080},10,[1082,1087,1092],{"type":24,"tag":55,"props":1083,"children":1084},{"style":62},[1085],{"type":30,"value":1086},"        $builder->",{"type":24,"tag":55,"props":1088,"children":1089},{"style":181},[1090],{"type":30,"value":1091},"setFirstResult",{"type":24,"tag":55,"props":1093,"children":1094},{"style":62},[1095],{"type":30,"value":1096},"($offset)\n",{"type":24,"tag":55,"props":1098,"children":1100},{"class":57,"line":1099},11,[1101,1106,1111],{"type":24,"tag":55,"props":1102,"children":1103},{"style":854},[1104],{"type":30,"value":1105},"                ->",{"type":24,"tag":55,"props":1107,"children":1108},{"style":181},[1109],{"type":30,"value":1110},"setMaxResults",{"type":24,"tag":55,"props":1112,"children":1113},{"style":62},[1114],{"type":30,"value":1115},"($limit);\n",{"type":24,"tag":55,"props":1117,"children":1119},{"class":57,"line":1118},12,[1120],{"type":24,"tag":55,"props":1121,"children":1122},{"style":62},[1123],{"type":30,"value":1124},"    }\n",{"type":24,"tag":55,"props":1126,"children":1128},{"class":57,"line":1127},13,[1129],{"type":24,"tag":55,"props":1130,"children":1131},{"emptyLinePlaceholder":1050},[1132],{"type":30,"value":1053},{"type":24,"tag":55,"props":1134,"children":1136},{"class":57,"line":1135},14,[1137,1142,1147,1152],{"type":24,"tag":55,"props":1138,"children":1139},{"style":854},[1140],{"type":30,"value":1141},"    return",{"type":24,"tag":55,"props":1143,"children":1144},{"style":62},[1145],{"type":30,"value":1146}," $builder->",{"type":24,"tag":55,"props":1148,"children":1149},{"style":181},[1150],{"type":30,"value":1151},"getQuery",{"type":24,"tag":55,"props":1153,"children":1154},{"style":62},[1155],{"type":30,"value":1156},"();\n",{"type":24,"tag":55,"props":1158,"children":1160},{"class":57,"line":1159},15,[1161],{"type":24,"tag":55,"props":1162,"children":1163},{"style":62},[1164],{"type":30,"value":1043},{"type":24,"tag":33,"props":1166,"children":1167},{},[1168],{"type":30,"value":1169},"It is an optional parameter and won’t break anything. If you do a Shopware update, this would be gone, but as I was in the quest to speed up things for a one-time task, I just modified it in the core instead of finding a long-term solution.",{"type":24,"tag":33,"props":1171,"children":1172},{},[1173],{"type":30,"value":1174},"What this function does is calculating a modulo of 1000 on the media id and compares it with the batch id. So we basically have 1000 batches to process until all work is done.",{"type":24,"tag":33,"props":1176,"children":1177},{},[1178],{"type":30,"value":1179},"Now we only need to start all 1000 batches in a parallel manner. For doing so, I used the very helpful tool parallel - which is available in Linux:",{"type":24,"tag":33,"props":1181,"children":1182},{},[1183],{"type":30,"value":1184},"It starts 64 batches in parallel and continues its work until all 1000 batches are finished.",{"type":24,"tag":33,"props":1186,"children":1187},{},[1188,1190,1195],{"type":30,"value":1189},"And this is how it looks at ",{"type":24,"tag":792,"props":1191,"children":1192},{},[1193],{"type":30,"value":1194},"htop",{"type":30,"value":1196},":",{"type":24,"tag":44,"props":1198,"children":1200},{"className":169,"code":1199,"language":171,"meta":7,"style":7},"parallel -j 64 ./bin/console my:image:generate:thumbnails --batch ::: {0..999}\n",[1201],{"type":24,"tag":51,"props":1202,"children":1203},{"__ignoreMap":7},[1204],{"type":24,"tag":55,"props":1205,"children":1206},{"class":57,"line":58},[1207,1212,1217,1222,1227,1232,1237,1242],{"type":24,"tag":55,"props":1208,"children":1209},{"style":181},[1210],{"type":30,"value":1211},"parallel",{"type":24,"tag":55,"props":1213,"children":1214},{"style":282},[1215],{"type":30,"value":1216}," -j",{"type":24,"tag":55,"props":1218,"children":1219},{"style":282},[1220],{"type":30,"value":1221}," 64",{"type":24,"tag":55,"props":1223,"children":1224},{"style":186},[1225],{"type":30,"value":1226}," ./bin/console",{"type":24,"tag":55,"props":1228,"children":1229},{"style":186},[1230],{"type":30,"value":1231}," my:image:generate:thumbnails",{"type":24,"tag":55,"props":1233,"children":1234},{"style":282},[1235],{"type":30,"value":1236}," --batch",{"type":24,"tag":55,"props":1238,"children":1239},{"style":186},[1240],{"type":30,"value":1241}," :::",{"type":24,"tag":55,"props":1243,"children":1244},{"style":186},[1245],{"type":30,"value":1246}," {0..999}\n",{"type":24,"tag":1248,"props":1249,"children":1254},"img",{"alt":1194,"aspect-ratio":1250,"height":1251,"object-fit":1252,"src":1253},"2",300,"contain","/blog/htop.png",[],{"type":24,"tag":25,"props":1256,"children":1258},{"id":1257},"finally-all-the-work-is-now-finished-in-35-hours-instead-of-80",[1259],{"type":30,"value":1260},"Finally, all the work is now finished in 3.5 hours instead of 80.",{"type":24,"tag":586,"props":1262,"children":1263},{},[1264],{"type":30,"value":590},{"title":7,"searchDepth":273,"depth":273,"links":1266},[1267,1268,1269],{"id":749,"depth":273,"text":752},{"id":777,"depth":273,"text":780},{"id":1257,"depth":273,"text":1260},{"_path":617,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":618,"description":619,"author":10,"image":620,"releaseDate":621,"blogCategories":1271,"articleTags":1272,"tags":1273,"body":1274,"_type":596,"_id":629,"_source":598,"_file":630,"_stem":631,"_extension":601},[14,15],[624,625],[627,628],{"type":21,"children":1275,"toc":1337},[1276,1282,1303,1308,1313,1319,1332],{"type":24,"tag":25,"props":1277,"children":1279},{"id":1278},"what-is-this",[1280],{"type":30,"value":1281},"What is this?",{"type":24,"tag":33,"props":1283,"children":1284},{},[1285,1287,1293,1295,1301],{"type":30,"value":1286},"This wrapper serves as a simple and cost-effective bridge that translates calls from the\n",{"type":24,"tag":764,"props":1288,"children":1290},{"href":1289},"https://docs.developer.amazonservices.com/en_US/dev_guide/index.html",[1291],{"type":30,"value":1292},"Amazon Marketplace Web Service (MWS)",{"type":30,"value":1294},"\nAPI to ",{"type":24,"tag":764,"props":1296,"children":1298},{"href":1297},"https://developer-docs.amazon.com/sp-api",[1299],{"type":30,"value":1300},"Selling Partner (SP) API",{"type":30,"value":1302}," calls. It was specifically developed for customers with outdated systems and limited\nbudgets, for whom a complete redevelopment of the Amazon interface would be economically unfeasible.\nThe wrapper allows these customers to continue their business even after the MWS interface is discontinued.",{"type":24,"tag":33,"props":1304,"children":1305},{},[1306],{"type":30,"value":1307},"Technically, the project is based on a Docker container that functions as a web server. This server receives MWS API calls,\ntranslates them into SP API calls, and converts the responses from the SP API back into the format of the MWS API.",{"type":24,"tag":33,"props":1309,"children":1310},{},[1311],{"type":30,"value":1312},"The idea for this wrapper originated from a proof of concept, with the aim to verify whether the interfaces are\nsufficiently similar to enable a direct translation. Despite concerns that the devil might be in the details,\nthe calls proved to be relatively simple to translate. It was only necessary to adjust some feed names\nand occasionally make two API calls to gather all the necessary information.\nOverall, the project has been surprisingly trouble-free.",{"type":24,"tag":25,"props":1314,"children":1316},{"id":1315},"download",[1317],{"type":30,"value":1318},"Download",{"type":24,"tag":33,"props":1320,"children":1321},{},[1322,1324,1330],{"type":30,"value":1323},"Interested? The project has been published as open-source on GitHub:\n",{"type":24,"tag":764,"props":1325,"children":1327},{"href":1326},"https://github.com/bhelm/Amazon-MWS-SP-Wrapper",[1328],{"type":30,"value":1329},"Amazon-MWS-SP-API-Wrapper",{"type":30,"value":1331},". This offer is aimed at developers\nwho also see the wrapper approach as the most efficient solution and want to contribute to the open-source community.",{"type":24,"tag":33,"props":1333,"children":1334},{},[1335],{"type":30,"value":1336},"If you need support integrating it into your application or if adjustments or extensions to the wrapper\nare necessary, we are happy to help. Since the wrapper is currently used by only a limited number of customers,\nit covers only part of the possible API calls and parameters and is by no means a \"complete solution\".",{"title":7,"searchDepth":273,"depth":273,"links":1338},[1339,1340],{"id":1278,"depth":273,"text":1281},{"id":1315,"depth":273,"text":1318},{"_path":636,"_dir":637,"_draft":6,"_partial":6,"_locale":7,"slug":10,"teams":1342,"primaryTeam":640,"firstName":641,"lastName":642,"prefixTitle":7,"suffixTitle":7,"education":1343,"executiveRole":648,"role":1345,"workingSince":657,"inTheCompanySince":658,"techSkills":1346,"skills":1363,"projects":1370,"contactDetails":1376,"_image":734,"image":735,"_id":736,"_type":737,"title":738,"_source":637,"_file":739,"_stem":740,"_extension":737},[639,640],[1344],[645,646,647],[650,651,640,652,653,654,655,656],[1347,1348,1349,1350,1351,1352,1353,1354,1355,1356,1357,1358,1359,1360,1361,1362],{"name":661,"level":662,"icon":663},{"name":665,"level":662,"icon":666},{"name":668,"level":662},{"name":670,"level":662,"icon":671},{"name":673,"level":662},{"name":675,"level":662,"icon":676},{"name":678,"level":662},{"name":680,"level":662},{"name":682,"level":662},{"name":684,"level":662,"icon":685},{"name":687,"level":688,"icon":689},{"name":691,"level":688,"icon":692},{"name":694,"level":688,"icon":695},{"name":697,"level":688,"icon":698},{"name":700,"level":688,"icon":701},{"name":703,"level":688,"icon":704},[1364,1365,1366,1367,1368,1369],{"name":707,"level":662},{"name":709,"level":662},{"name":711,"level":662},{"name":713,"level":662},{"name":715,"level":662},{"name":717,"level":688},[1371,1372,1373,1374,1375],{"project":720,"position":721},{"project":723,"position":721},{"project":725,"position":721},{"project":727,"position":728},{"project":730,"position":721},{"eMail":732,"phone":733,"visibility":303},[1378,1392],{"_path":1379,"_dir":1380,"_draft":6,"_partial":1050,"_locale":7,"name":1381,"slug":1380,"text":1382,"hoverText":1383,"image":1384,"customer":1381,"tags":1385,"_id":1387,"_type":1388,"title":1389,"_source":598,"_file":1390,"_stem":1391,"_extension":1388},"/en/portfolio/pixelx/_teaser","pixelx","PixelX","IT Security with Precision and Expertise","For PixelX, we conducted a targeted security analysis where, thanks to our deep technical understanding, we were able to identify a critical SQL injection vulnerability. With minimal time investment, we achieved maximum security gain.","/images/portfolio/pixelx/pixelx_secured.png",[1386,19],"security","common:en:portfolio:9010.pixelx:_teaser.yaml","yaml","Teaser","en/portfolio/9010.pixelx/_teaser.yaml","en/portfolio/9010.pixelx/_teaser",{"_path":1393,"_dir":1394,"_draft":6,"_partial":1050,"_locale":7,"name":1395,"slug":1394,"text":1396,"hoverText":1397,"image":1398,"customer":1395,"tags":1399,"_id":1400,"_type":1388,"title":1389,"_source":598,"_file":1401,"_stem":1402,"_extension":1388},"/en/portfolio/slimspots/_teaser","slimspots","SlimSpots","Processing large amounts of data in real-time","For SlimSpots, a global provider of ad marketing solutions, we developed a highly scalable infrastructure that enables the processing of trillions of data records in real-time.","/images/portfolio/slimspots/slim_spots_prtfolio.png",[711,19],"common:en:portfolio:9020.slimspots:_teaser.yaml","en/portfolio/9020.slimspots/_teaser.yaml","en/portfolio/9020.slimspots/_teaser",1782284053725]