ISCSI Storage ETW tracing
Andera Swearengin
The main reason to separate LAN and SAN traffic is that you want to make sure your storage network cannot clog up at all events. If it did, it would rapidly cause I/O errors, in turn causing data loss and even corruption. A (very) low volume of stray traffic isn't anything to really worry about.
Ever since the crash, once I get to the point I have about 30 iSCSI shared storage VMs running I am unable to start anymore and start to run into issues. (This is nowhere near the number of VMs we had running before the crash.) It does not appear to be any specific VMs, or Specific XenServer host, or specific iSCSI LUN. Logs on both the XenHosts and storage repository seem to indicate iSCSI connection issues. My research indicates that these type of issues seem to fall into X categories:
I could not find what 1021 means only 1020 which pointed me in the direction of network issues. Do you think that the 1021 'target is rejecting the connection' is purely a storage repository issue or could it also be the XenServer hosts causing the SR to react that way?
client IP = 172.18.1.14 Storage IP = 172.18.1.15 All on same switch and same VLAN. First of all the client sends a read LUN request in frame 8. This looks fine. But I'm not sure why he keeps sending ACKS. Then in frame 17 the storage sends back the data requested in the read. However, there's a few things to note here. 1) This ia an ACK to frame 8. 2) The PUSH bit is set. 3) The frame is not a full jumbo frame of 9014 bytes.
Client sends a read LUN request. Next frame from storage is an ACK only from the storage (60 byte packet) no data Next frame from the storage is read data requested with full 9000 bytes. ACK bit is set but no push
Final verdict: unfortunately, your trace is useless and cannot be used to diagnose any trouble of the iSCSI connection. Your capturing device was not fast enough to record the jumbo data frames to disk as they appeared, and only was able to write the small ACKs and the occasional non-full jumbo frame to disk. You cannot diagnose iSCSI - or any high bandwith shared storage protocol - with capture hardware that doesn't at least write 120MB/s to disk. Because that's what a full Gigabit link will slam your capturing NIC with if it is doing full throttle in one direction. It gets worse if you capture a Gigabit link with full throttle in both directions, because then you need to write about 240MB/s. So unless you captured with a fast RAID disk configuration or an SSD setup you're not fast enough. A laptop with a single non-SSD disk will never be fast enough for this if the storage system gets going at full speed.
You can verify whether the connection is accessible by using the ping command. You can also verify by connecting to iSCSI port of the storage device by using the telnet command to ensure that the iSCSI service is available. The default port is 3260.
This section lists the functions, callbacks, macros, structures, and enumerations that can be used to develop various Windows storage drivers. The list of header files that contain these declarations and definitions follows. All headers are included in the Windows Driver Kit (WDK).
AWS Storage Gateway is a hybrid cloud storage service that gives you on-premises access to virtually unlimited cloud storage. Storage Gateway provides a standard set of storage protocols such as iSCSI, SMB, and NFS, which allow you to use AWS storage without rewriting your existing applications. It provides low-latency performance by caching frequently accessed data on premises, while storing data securely and durably in Amazon cloud storage services. Storage Gateway optimizes data transfer to AWS by sending only changed data and compressing data. Storage Gateway also integrates natively with Amazon S3 and Amazon FSx for Windows File Server cloud storage, which makes your data available for in-cloud processing, AWS Identity and Access Management (AWS IAM) for securing access management to services and resources, AWS Key Management Service (AWS KMS) for encrypting data at rest in the cloud, Amazon CloudWatch for monitoring, and AWS CloudTrail for logging account activity.
The Volume Gateway provides block storage to your on-premises applications using iSCSI connectivity. Data on the volumes is stored in Amazon S3 and you can take point-in-time copies of volumes that are stored in AWS as Amazon EBS snapshots. You can also take copies of volumes and manage their retention using AWS Backup. You can restore EBS snapshots to a Volume Gateway volume or an EBS volume.
You use the AWS Management Console to download the virtual appliance gateway or purchase the hardware appliance, configure storage, and manage and monitor the service. The gateway connects your applications to AWS storage by providing standard storage interfaces. It provides transparent caching, efficient data transfer, and integration with AWS monitoring and security services.
Amazon S3 File Gateway presents a file-based interface to Amazon S3, which appears as a network file share. It enables you to store and retrieve Amazon S3 objects through standard file storage protocols. File Gateway allows your existing file-based applications or devices to use secure and durable cloud storage without needing to be modified. With S3 File Gateway, your configured S3 buckets will be available as Network File System (NFS) mount points or Server Message Block (SMB) file shares. Your applications read and write files and directories over NFS or SMB, interfacing to the gateway as a file server. In turn, the gateway translates these file operations into object requests on your S3 buckets. Your most recently used data is cached on the gateway for low-latency access, and data transfer between your data center and AWS is fully managed and optimized by the gateway. Once in S3, you can access the objects directly or manage them using S3 features such as S3 Lifecycle Policies and S3 Cross-Region Replication (CRR). You can run S3 File Gateway on-premises or in EC2.
Volume Gateway provides an iSCSI target, which enables you to create block storage volumes and mount them as iSCSI devices from your on-premises or EC2 application servers. The Volume Gateway runs in either a cached or stored mode.
AWS Storage Gateway provides a set of features that enable you to effectively leverage AWS storage within your existing applications and workflows. It provides a standard set of protocols such as iSCSI, SMB and NFS, which allow you to use your existing applications without any changes. Through its local cache, the gateway provides low-latency access to recently used data. The gateway optimizes data transfer to AWS storage, such as optimization of transfer through intelligent buffering, upload management to address network variations, and bandwidth management. The gateway provides you an effective mechanism to store data in AWS across the range of storage services most suitable for your use cases. The gateway is easy to deploy and can use your existing virtual infrastructure and hypervisor investments, or can be installed in your data center or remote offices as a hardware appliance. The gateway software running as a VM or on the hardware appliance is stateless, allowing you to easily create and manage new instances of your gateway as your storage needs evolve. Finally, the service integrates natively into AWS management services such as Amazon CloudWatch, AWS CloudTrail, AWS Key Management Service (KMS), and AWS Identity and Access Management (IAM).
Amazon S3 File Gateway supports Amazon S3 Standard, S3 Intelligent-Tiering, S3 Standard - Infrequent Access (S3 Standard-IA) and S3 One Zone-IA. For details on storage classes, refer to the Amazon S3 documentation. You configure the initial storage class for objects that the gateway creates, and then you can use bucket lifecycle policies to move files from Amazon S3 to Amazon S3 Glacier. If an application attempts to access a file/object stored through Amazon File Gateway that is now in Amazon S3 Glacier, you will receive a generic I/O error.
Files are stored as objects in your S3 buckets and you can configure the initial storage class for objects that File Gateway creates. There is a one-to-one relationship between files and objects, and you can configure the initial storage class for objects that Amazon S3 File Gateway creates.
You can use S3 lifecycle policies to change an object's storage tier or delete old objects or object versions. In the case of objects deleted by lifecycle policy, you will need to enable the periodic cache refresh feature or call the RefreshCache API to reflect these changes to your NFS clients.
Local disk storage on the gateway is used to temporarily hold changed data that needs to be transferred to AWS, and to locally cache data for low-latency read access. File Gateway automatically manages the cache maintaining the most recently accessed data based on client read and write operations. Data is evicted from the cache only when space is needed to store more recently used data.
Many on-premises desktop applications are latency-sensitive, which may cause delays to your end users and slow performance when they are directly accessing files in AWS from remote locations. Additionally, allowing large numbers of users to directly access data in the cloud can cause congestion on your shared bandwidth resources such as AWS Direct Connect links. Amazon FSx File Gateway allows you to use Amazon FSx for Windows File Server for these workloads, and help replace your on-premises storage with fully managed, scalable, and highly reliable file storage in AWS without impacting your applications or network.
The Snowball Edge Storage Optimized device provides 80 terabytes of usable block storage or object storage and can migrate that amount of tape data to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive.
You get 11 9s of data durability, fixity checks by AWS on a regular basis, data encryption, right data when you restore, and cost savings, when storing virtual tapes in AWS using Tape Gateway with S3 Glacier Deep Archive compared to warehousing physical tapes offsite. First, all virtual tapes stored in S3 Glacier Deep Archive are replicated and stored across at least three geographically-dispersed Availability Zones, protected by 11 9s of durability. Second, AWS performs fixity checks on a regular basis to confirm your data can be read and no errors have been introduced. Third, all tapes stored in S3 Glacier Deep Archive are protected by S3 Server Side Encryption using default keys or your KMS keys. In addition, you also avoid physical security risk associated with tape portability. Fourth, compared to the experience of warehousing tapes offsite where you may receive an incorrect or broken tape during restore, with Tape Gateway, you always get correct data. Finally, you can save in monthly storage costs when storing your data in S3 Glacier Deep Archive compared to warehousing tapes offsite.
dd2b598166