Thanks for reaching out. Unfortunately we cannot guarantee compatibility with third-party libraries like aws-vault. To investigate a boto3 issue here we would need a code snippet to reproduce the issue, in addition to debug logs (with sensitive info redacted) which you can get by addingboto3.set_stream_logger('')to your script.

aws-vaultis just a convenient way to generate the necessary ENVs to authorizing withboto3and the same problem can be reproduced with any token that expires. I'll grab logs of it happening.

Here's a minimal reproducer (and I believe the simplest example of usingBaseSubscriberandon_donethat's possible) and I've attached the logs from running it with an expired token:

import logging
from io import BytesIO

from boto3 import set_stream_logger
from boto3.s3.transfer import BaseSubscriber
from boto3.session import Session
from s3transfer.manager import TransferConfig, TransferManager

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

set_stream_logger( "" )


class CheckDone(BaseSubscriber):
def on_done(self, future, **kwargs) -> None:
try:
res = future.result()
logger.info( "Upload worked: %s", res)
except Exception as e:
logger.error( "Upload failed: %s", e)


def _main() -> None:
session = Session()
s3_tm = TransferManager(session.client( "s3" ), TransferConfig(max_request_concurrency=3))
s3_tm.upload(
BytesIO(b "testing" ),
"test_bucket",
"s3transfer_test.txt",
subscribers=[CheckDone( "s3transfer_test.txt" )],
)
s3_tm.shutdown()
logger.info( "Done" )


if __name__ == "__main__":
_main()

s3transfer_test.log

Thanks for following up. We don't recommend using s3transfer directly, as noted in theREADME:

This project is not currently GA. If you are planning to use this code in production, make sure to lock to a minor version as interfaces may break from minor version to minor version. For a basic, stable interface of s3transfer, try the interfaces exposed inboto3

Have you tried using the Boto3uploadmethods for S3? You can handle any errors asdocumented hereor handle events likeafter-call-error.

Sorry, I pulled from the wrongimportwhen putting together that minimal reproducer. Here's it with theimportfromboto3:

import logging
from io import BytesIO

from boto3 import set_stream_logger
from boto3.s3.transfer import BaseSubscriber, TransferConfig, TransferManager
from boto3.session import Session

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

set_stream_logger( "" )


class CheckDone(BaseSubscriber):
def on_done(self, future, **kwargs) -> None:
try:
res = future.result()
logger.info( "Upload worked: %s", res)
except Exception as e:
logger.error( "Upload failed: %s", e)


def _main() -> None:
session = Session()
s3_tm = TransferManager(session.client( "s3" ), TransferConfig(max_concurrency=3))
s3_tm.upload(
BytesIO(b "testing" ),
"test_bucket",
"s3transfer_test.txt",
subscribers=[CheckDone( "s3transfer_test.txt" )],
)
s3_tm.shutdown()
logger.info( "Done" )


if __name__ == "__main__":
_main()

That snippet is still using the TransferManager directly which isn't recommended:

Thanks for following up. We don't recommend using s3transfer directly, as noted in theREADME:

This project is not currently GA. If you are planning to use this code in production, make sure to lock to a minor version as interfaces may break from minor version to minor version. For a basic, stable interface of s3transfer, try the interfaces exposed inboto3

Have you tried using the Boto3uploadmethods for S3? You can handle any errors asdocumented hereor handle events likeafter-call-error.

Then what is the recommended way to upload multiple files in parallel in a separate thread usingboto3?

This is theonly information I could find about uploading in paralleland it looks like I'm doing a simpler version of what that code shows, so is there a better way?

Hi@daveisfera— we recommend usingupload_fileorupload_fileobj.And here is thedocumentationon multithreading/multiprocessing. Although the S3Transfer code you shared is public it's still recommended to use Boto3 directly.

Also looking back at thelogsyou shared, it looks like you'reon_donefunction is being called and the exception is logged (ERROR:__main__:Upload failed:...).

Hi@daveisfera— we recommend usingupload_fileorupload_fileobj.And here is thedocumentationon multithreading/multiprocessing. Although the S3Transfer code you shared is public it's still recommended to use Boto3 directly.

I'm glad to switch but that documentation doesn't seem to provide information on how to actually perform the uploads in another thread. It also mentions events but I don't see how to receive an event notification likeon_done,so is there detailed information on how to use it?

Also looking back at thelogsyou shared, it looks like you'reon_donefunction is being called and the exception is logged (ERROR:__main__:Upload failed:...).

You are correct. I'll check and see if I can get logs from when the callback doesn't happen

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

Still waiting for guidance on how to do the upload in parallel and receive notifications of when it completes or has an error

I think we were still waiting on logs where the error referencedheredoesn't occur. Going back to the Boto3events,wouldn't theafter-callorafter-call-errorbe what you're looking for?

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

I think we were still waiting on logs where the error referencedheredoesn't occur.

I can reproduce the issue with the original example I sent but not the updated one that imports fromboto3rather thans3transferso I think this was related to that and probably not an actual issue since the original example isn't a supported use case.

Going back to the Boto3events,wouldn't theafter-callorafter-call-errorbe what you're looking for?

It appears that's a general callback and not tied to a specific object/call so is there an example of how to link those calls back to the original call?

Basically, thes3transferinterface appears to be a better/simpler way to handle uploads/downloads in parallel and is documented well enough to get it working. I'm definitely fine with switching to a different interface, but would need guidance on how to do that because I'm not seeing the info in the documentation on how to do that.