25 Replies Latest reply on Apr 28, 2016 11:06 AM by ivan rischmiller

    Blueprint 5.1 & load balance

    Wissam Yassin Wayfarer

      Hi All,

       

      I am planning to integrate a load balancer with blueprint 5.1 . The reason for that is I want the print queues to be distrubited equally to the collectors ( 6 collectors).

       

      Did any one implement that before. If yes, how will be the end user configuration ? I mean how to map a network printer to the end user ?

       

       

       

      Appreciate your sharing of that.

       

       

       

      Thanks

       

        • Re: Blueprint 5.1 & load balance
          Scott Olswold Guide

          Wassim,

           

          Please review this post:

           

          Pharos Blueprint Enterprise and Load Balancing Solutions

           

          Thank you,

          Scott

            • Re: Blueprint 5.1 & load balance
              Wissam Yassin Wayfarer

              Hi Scott,

               

              I saw this post after submitting mine. Sorry

                • Re: Blueprint 5.1 & load balance
                  Wissam Yassin Wayfarer

                  Hi Scott,

                   

                  I have applied the concept of NLB and everything looked fine. However, when we tried to connect  printer , it was giving me an error message ( common mentioned error message )

                  "    Microsoft Windows 2008, 2008 R2, 2012, and 2012 R2 servers may need a "DNSOnWire" configuration (see

                    • http://support.microsoft.com/kb/979602) to avoid "Windows couldn't connect to the printer. Check the printer name and try again. If this is a network printer, make sure that the printer is turned on, and that the printer address is correct." error messages on Windows clients. "

                  Even after applying the fix from Microsoft , still the same. We have contacted Microsoft and their reply was the following :

                   

                  NLB that is configured is a failover cluster offer high performance in environment in which each request from a client is stateless, and there is no in-memory application state to maintain. You cannot implement this configuration together with a print server. This is becasue print servers maintain losts of state information for each printer and client with which they are communicating. Print jobs may span several communication requests. Therefore, all requests must be handled by the same node.

                   

                  http://support.microsoft.com/kb/979602

                   

                        Enable Client Side Rendering (CSR) on as many clients as possible

                  This reduces the work the server has to perform on each print job by rendering the print job on the individual clients before it sends the job to the server. Windows Vista-based and newer computers support CSR

                           Increase the resources on the server

                  If a server is the cause of poor performance, it will most likely encounter problems with insufficient processor power, or memory. Increasing the capabilities of those components should improve the responsiveness of the server.

                   

                  http://support.microsoft.com/kb/954420/en-us/

                   

                  http://blogs.technet.com/b/askperf/archive/2008/10/03/windows-printing-and-network-load-balancing.aspx

                   

                  Any advise !!

                   

                  Thanks

                   

                   


                    • Re: Blueprint 5.1 & load balance
                      Scott Olswold Guide

                      Wissam,

                       

                      Thanks for reminding me to make a call-out in the article:

                       

                      WARNING
                      Beginning with Windows Server 2008, Microsoft introduced a feature called Network Load Balancing, or NLB. NLB is not supported for print servers, nor does Pharos Blueprint work within its framework. In order to load balance a Blueprint system, a third-party provider must be implemented.

                       

                      Microsoft's Network Load Balancing looks attractive because it is free (well, free in that it is included with the standard license for Windows Server), and for tasks like web serving, it works amazingly well. I have seen (and tried) a couple of blog posts that discuss how to make print sharing work (it does) within NLB (surprisingly), but I have major reservations about pinning the success of any implementation upon a hacked technique that could be broken through the installation of a Microsoft KB patch.

                       

                      Regards,

                      Scott

                    • Re: Blueprint 5.1 & load balance
                      Nikolay Karetnikov Navigator

                      Hello! Sorry, Wissam, can you outline which tech do you use for nlb?

                        • Re: Blueprint 5.1 & load balance
                          Wissam Yassin Wayfarer

                          Hello Nik,

                           

                          To be honest we did not user NLB .  We were planning to use either Microsoft LB or netscaler load balancer.

                          We still have win 2008 R2 with Blueprint 5.1. We might think of adding load balancer with the next version of Pharos.

                           

                          However, we did a work around. ( Manual load balancer ). We created alias name for each server. In case of one server went down, we point the alias name to another server. Also we dedicated a server for each 3 department based on the printing load they have. ( 2000 users we have in the company ).

                           

                          Thanks

                      • Re: Blueprint 5.1 & load balance
                        Ashraf Abu Sallam Newbie

                        Thank you for explanations.

                        • Re: Blueprint 5.1 & load balance
                          Scott Olswold Guide

                          I modified this document on 26 April 2016 to include some warning information about using Windows servers in a load-balanced print queue sharing scenario (mostly because Microsoft doesn't support using the Windows Spooler in this way), as well as some additional information about the use of a SAN (Subject Alternate Name) certificate rather than a wildcard certificate.

                            • Re: Blueprint 5.1 & load balance
                              ivan rischmiller Wayfarer

                              Cheers Scott. I just re-read the doc and noted the updates.

                               

                              Someday maybe I’ll mess around with the MS NLB and printing, but not today

                               

                              The F5s were already in place and luckily we were able to leverage them. It remains to be seen what throughput and performance is once we have all 30,000 users going through the Load Balanced environment, but the architects assure me the capacity is there We expect to have all end users fully changed over by the end of June 2016, I’ll be sure to report back on our success…

                               

                              I will post my experience bringing the environment up soon.

                               

                              Thanks

                              Ivan

                                • Re: Blueprint 5.1 & load balance
                                  Scott Olswold Guide

                                  Ivan,

                                  Microsoft's Network Load Balancing implementation is useless for print serving; it is wildly unstateful in its implementation (I can barely decipher how it manages to wrangle HTTPS sessions; it must rely on cookies) and you will end up wondering if you can charge Microsoft for your wasted time.

                                   

                                  As far as throughput and performance goes: implementing f5 GTM/LTM requires centralizing your print server fleet to data centers and away from office-based print servers. While f5 BIG-IP is licensed based on maximum throughput (and it can handle a lot), and it is true that 1KB is 1KB is 1KB, the distance packets have to travel increases, the connection has the potential to cross an unstable or throttled WAN connection, and can, for some users, create an almost unbearable workflow where some tasks and operations (mostly involving Microsoft Office applications and their obsessive desire to issue OpenPrinter and GetPrinterInfo calls to the active/default print queue) take a lot longer than they used to. Some of your more sensitive users might end up like this guy:

                                   

                                  Angry Worker Destroy Computer - YouTube

                                   

                                  -Scott

                                    • Re: Blueprint 5.1 & load balance
                                      ivan rischmiller Wayfarer

                                      Ohhhh we have soooo many stories I could share about people who are having trouble coming to terms with secure release and the performance differences, let alone the load balanced environment. I won’t even mention going from the ricoh 45 ppm cadillacs to HP 32 ppm if you’re lucky k cars. I know everyone has their stories, but we have had some real doozies. The lady who talked about the print sit and spin class was a jewel.

                                       

                                      If this load balanced environment introduces anything more than ten seconds difference we will never hear the end of it.

                                       

                                      So far though, I have been unable to measure any significant difference between end users printing directly to a collector or printing to the load balanced queue. Unless a collector that is recognized by the load balancer is not available. Then all hell breaks loose. That issue looks like we will not be able to overcome anytime soon, but we are being careful to include that caveat in the service description so at least we won’t pay penalties for it.

                                       

                                      So much for doing maintenance during business hours tho

                                       

                                      ivan

                                        • Re: Blueprint 5.1 & load balance
                                          Scott Olswold Guide

                                          The "unavailable" Collector issue is dealt with at the bottom of my Load Balancing post, in the Troubleshooting section. You basically have to scalpel out the offending Collector data from the "psbprint" database.

                                            • Re: Blueprint 5.1 & load balance
                                              ivan rischmiller Wayfarer

                                              Hi Scott.

                                               

                                              Thanks for this info!!! Great stuff!

                                               

                                              I have not seen the driver not loading correctly at the workstation yet. Our testing has so far shown the driver install to work correctly as long as the end user doesn’t interfere until it is complete… We don’t have any authority or visibility into maintaining the workstations, but I will certainly keep this information handy and share the hotfix information with the workstation folks to try and get ahead of any issues related.

                                               

                                              Is it x86 only? I don’t think we have any x86 workstations beyond maybe a handful of test and dev machines. Everything went x64 on the last refresh.

                                               

                                              I have been adamant that when we change the end users to our load balanced queues, the existing queue(s) on the workstation are removed, then the driver(s) are removed, then the new queue is added and the workstation allowed to pick up the driver again from the server(s). This appears to work well and the workstation team has come up with an addition to the log in script that almost does this. They seem to be waffling on removing the driver :\  And yes we have communicated our concern about tying this into a log in script, but this the process they have chosen. My recommendation was to just apply the script in bulk during the change window, but they rejected that. They are the workstation experts, so we let them control how they apply changes. I hope it doesn’t cause issues.

                                               

                                              I noticed the bit about carving out the unavailable collector, and in a situation where a collector or collectors may be unavailable for an extended period, we would certainly use the process. In fact I will try to test it before we are at full production. If I don’t I can still test within our test environment, but those collectors still point to the production analyst. Anyhow I’ll give it a shot.

                                               

                                              Thanks again, much appreciated.

                                               

                                              BTW looking forward to seeing some information about the 2017 pharos conference, I’m hoping there is enough Blueprint 5.2 training to be able to justify to my managers why they should send me

                                               

                                              Ivan

                                              • Re: Blueprint 5.1 & load balance
                                                ivan rischmiller Wayfarer

                                                Sorry for the SPAM Scott, I see the hotfix is available for x86 and x64. I’ll chat with the workstation people and find out if this hot fix has already been applied and if not get them to roll it out.

                                                 

                                                Cheers

                                                Ivan

                                                • Re: Blueprint 5.1 & load balance
                                                  ivan rischmiller Wayfarer

                                                  Sorry x2 for the spam lol…

                                                   

                                                  Scott you wouldn’t happen to have a KB article for the incomplete driver install issue would you?

                                                   

                                                  I found https://support.microsoft.com/en-us/kb/2864755 but I’m not sure that it is the one.

                                                   

                                                  I’m hoping these hotfixes have been rolled into one of the gajillion windows 7 updates …

                                                   

                                                  Cheers

                                                  Ivan

                                                    • Re: Blueprint 5.1 & load balance
                                                      Scott Olswold Guide

                                                      Ivan,

                                                      NP. That's the article that I landed on.

                                                      -Scott

                                                        • Re: Blueprint 5.1 & load balance
                                                          ivan rischmiller Wayfarer

                                                          Good Morning Scott.

                                                           

                                                          So as with most KB articles, there is a certain level of not enough practical information and lots of technical mumbo jumbo lol.

                                                           

                                                          All I really need to know is – Do we need to apply this hotfix to the server AND the client workstation or is it good enough to have it applied in one spot (the server OR the workstation) If you have any insight into this please feel free to share

                                                           

                                                          It would appear that our 2008 R2 sp1 print servers are eligible for these hotfixes, our 2012 R2 servers do not require it. All 24 of our collectors are 2012 R2.

                                                           

                                                          I’m guessing the workstations have it already, but I don’t know and may never know

                                                           

                                                          I did communicate this information to the Team Lead of the group that is responsible for the workstations however I will be very surprised if I get a response. Different company and silos and all that.

                                                           

                                                          Anyhow, thanks again for this info, much appreciated.

                                                          Ivan

                                          • Re: Blueprint 5.1 & load balance
                                            ivan rischmiller Wayfarer

                                            Hello.

                                             

                                            We have created a load balanced environment using F5 load balancers. We had much help from the technical staff at Pharos. It is operational for our Secure Release end users. We use a virtual IP (V.IP) address which points to the load balancer. The load balancer is then configured to be aware of the static IP addresses of the collectors. Currently the F5's are set to round robin which appears to work currently. We use DNS records to translate from our DNS queue name to the V.IP. We have two V.IPs, one is for production and one is for test. I highly recommend having a test environment. our test environment has four collector servers in it, but point to our production Analyst. We have found one item that can present a performance change at the end user experience. When one or more collector servers within the load balanced (LB) environment is unavailable, print performance is degraded by about 40 seconds total at the end user. Pharos is aware of this issue and working at possibly integrating a solution. With the exception of that item, our environment works flawlessly. We have a total of 12 collectors within the LB environment and print about 2 million jobs per month. We service about 30,000 users.

                                             

                                            We are able to provide two single queues to our end users.

                                             

                                            We have:

                                            \\dnsname\PullPrint

                                            \\dnsname\PullPrintPS

                                             

                                            We have also tested pointing our MFP fleet to the dns name that points to the load balanced virtual IP address within the HP/Pharos iMFP solution 4.2.6 and it works without issue. This means we are not required to point our devices specifically to a collector. This has several advantages including a reduction in fleet maintenance activity if we change or remove a collector, and means that we don't have to try and manage how many devices are pointed to a single collector. We are using WebJetAdmin to change the configuration of our print device fleet HP /Pharos iMFP solution in bulk. All of our print devices are HP printers. We are still in process of moving our printer fleet to dns, so if we see any issues with this I can post here.

                                             

                                            What was quite interesting was transitioning from a traditional environment where you have a queue or queues on a single collector and the end user must add the queue \\servername\queuename and the print device must be pointed to a collector to the load balanced environment. We had to do this with zero impact to the end user or the secure release service.

                                             

                                            If anyone is interested, I can post further details. I basically have a worksheet that details each change required to the traditional environment step by step.

                                              • Re: Blueprint 5.1 & load balance
                                                Scott Olswold Guide

                                                Ivan,

                                                When you were preparing the servers, did you use the instructions at Pharos Blueprint Enterprise and Load Balancing Solutions? If so, were there any changes that you had to make to the process?

                                                Thanks,

                                                Scott

                                                  • Re: Blueprint 5.1 & load balance
                                                    ivan rischmiller Wayfarer

                                                    HI Scott. That document was part of our process. I created a more focused technical doc that detailed adding the loopback and making sure to use a SAN cert (so that we could still allow users to print directly to individual collectors as well as use the load balanced solution). I basically combined information from a couple of Pharos docs and some emails into a basic worksheet. If there is any interest, I could post exactly what I did to make things work... The doc I made is only a few pages. And I could add a little more info to my summary of our environment above...

                                                      • Re: Blueprint 5.1 & load balance
                                                        Scott Olswold Guide

                                                        Ivan,

                                                        Thank you for the offer. Others may benefit from your document, so feel free to share. When I crafted the original document, I kept some of the finer points out to protect the innocent (kind of like a MacGyver episode, where they didn't show you all of the ingredients for the bubble-gum explosive).

                                                        -Scott

                                                        • Re: Blueprint 5.1 & load balance
                                                          Scott Olswold Guide

                                                          Ivan,

                                                           

                                                          There's one other thing to watch out for. there's a bizarre behavior where the print driver is incompletely loaded. It looks on the surface that everything at the connection attempt works, but the printer driver load is actually incomplete. You'll see it by either going into Printing Preferences (you'll see none of the normal tabs; you'll see two tabs for a generic driver), or, for the brave, you can travel to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\Environments\Windows x64\Drivers\Version-3 and look at the contents of the key for the driver name. You'll see that the "Dependent Files" value is empty. The result for the end user is that there's a print, it looks like it succeeded, but there's nothing there to release (and no errors in any Blueprint log). At the server side, the job was received and summarily rejected by the server's Print Spooler. This happens in Windows 7 only, and can be fixed by applying the following hot fixes from Microsoft:

                                                           

                                                          •    469371_intl_x64_zip.exe << or "_x86_" for 32-bit OSes

                                                          •    478588_intl_x64_zip.exe << or "_x86_" for 32-bit OSes

                                                          •    479772_intl_x64_zip.exe << or "_x86_" for 32-bit OSes

                                                           

                                                          The defect is not related to the HP driver, but to a problem in Windows 7 where, when it connects to a Windows server share, it incorrectly identifies the server as a “down-level” (i.e., non-Windows) server and so gets only a generic Unidriver load (but says that it's the UPD or whatever). I suspect that going through two packet-shaping systems ends up muddying the waters for the Windows client. Once the hot fixes are applied, all shared queues experiencing the issue must be disconnected from the client, the driver removed, and the shares reconnected. Removing the driver is as straightforward as removing the subkey under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\Environments\Windows x64\Drivers\Version-3 that correspond with the driver (for example, HP Universal Printing PCL 6 (v5.8.0)).

                                                           

                                                          -Scott