ssl method doesn't work so well using resnet18 on imagenet

Hi thanks for the open source implementation of pruning, it really saves me a lot of time.
But when I try to modify the recipe in the examples/ssl/ssl_filter-removal.yaml to make it work in resnet on imagenet dataset. The result I get is not so well,  the sparsity is around 18%, but acc1 drop from 69% to 51%.
I try to figure out what is going on, but I cannot find the reason.
Does anyone get the same result? Or does someone work very well using ssl filter remove method on resnet18 in imagenet, could u share your recipe with me? I would appreciate that if someone can help me :)
Thanks.

Hi @Yifan122,
Regularization does not work out of the box, unfortunately.  In fact, I found it pretty hard to get to work well (see the comments at the top of a couple of the SSL YAML files) because you need to manually set the regularization strength for each layer.
How do you do that?  You need to start guessing and experimenting with different values.  It's even harder because the regularization strengths of one layer affect other layers (see my comments at the top of this schedule).
So I don't think you are doing something wrong necessarily, you just need to experiment more.  But perhaps SSL is not the method you want to use because of its challenges.  This field of model compression is new and changing and there are all kinds of algorithms, but I still haven't found an algorithm or method that works well and is practical for various workloads (classification, segmentation, language, etc.).  If you are a researcher, this is good news because there is lots to do.  If you are an engineer that wants to compress arbitrary networks, this is not so good news.
I hope this helped answer your question.
Cheers
Neta


grpc: standardize the cache interface of grpc async client.

For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md
Commit Message: Creating a grpc async client can be expensive. Caching and reusing grpc async client can save resources and improve stats consistency. This pr add a thread local cache of grpc async client as a member of grpc async client manager.
Additional Description:N/A
Risk Level:medium
Testing:add unit and integration tests
Docs Changes:N/A
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]#12598
[Optional Deprecated:]
[Optional API Considerations:]

@adisuissa @dmitri-d I think you folks might be interested in taking a pass on this from a control plane perspective.

/wait

I think we should cache by default. Whether you want to make a non-cachable configuration option I have less strong an opinion on, but principle-of-least-surprise points to caching as default (by construction as well; folks shouldn't have to hunt for the normal performance way of building a client).

/retest

/wait

needs main merge, as well as the integration test per above
/wait

/retest

Chaoqin: with this passing CI is this ready for review? Can you clean up the PR description? That is, fill out all non-optional fields, even if the answer is "N/A"?

/wait

Chaoqin: with this passing CI is this ready for review? Can you clean up the PR description? That is, fill out all non-optional fields, even if the answer is "N/A"?

Thanks, have filled up theses fields, it is ready for review.

/wait

/wait

/retest

/wait

Needs format fix. LGTM otherwise.

/retest

/retest

This needs a non-google review I think. @lizan  can you take a look?

Thanks @lizan . @chaoqin-li1123 can you merge main?

/wait

@jmarantz @htuch @chaoqin-li1123 sorry late to this PR, but I see a lot of comments about cached vs. uncached. From a skim of the PR I'm not quite sure why some users were converted to cached and some were not. Can we at minimum backfill TODO comments or other comments on every callsite using uncached to explain why that is happening? From reading it quickly it's confusing. Thank you.

@jmarantz @htuch @chaoqin-li1123 sorry late to this PR, but I see a lot of comments about cached vs. uncached. From a skim of the PR I'm not quite sure why some users were converted to cached and some were not. Can we at minimum backfill TODO comments or other comments on every callsite using uncached to explain why that is happening? From reading it quickly it's confusing. Thank you.

I think ultimately we want to use the cached version everywhere. But for now only the grpc client for ex_authz is default enabled, which is equivalent to the old behavior. We can open an issue as followup and clean this up in the following pull request.

I think ultimately we want to use the cached version everywhere. But for now only the grpc client for ex_authz is default enabled, which is equivalent to the old behavior. We can open an issue as followup and clean this up in the following pull request.

Please go through and add comments/TODOs on each of the usages when you open the issue. Thank you!

Yes, this was also my question but I think the position of wanting to derisk this by starting with ext_authz is fine modulo filing the issues.

Checking back in as we cut the release, what's the plan for flipping this true?

We finish the migration for ext_authz and ext_proc.


Error: The breadcrumb name must be a non-empty string.

After integrating bugsnag-php with OctoberCMS, whenever I access a specific backend update page that uses raw DB queries, I get error 'The breadcrumb name must be a non-empty string.'
This is the case with the bugsnag-php versions I tested ( 3.5.0 - 3.9.0 )
The only workaround I can come up with to avoid this error is:
replace the line: https://github.com/bugsnag/bugsnag-php/blob/master/src/Client.php#L294
$this->leaveBreadcrumb($report->getName(), Breadcrumb::ERROR_TYPE, $report->getSummary());
with:
$this->leaveBreadcrumb(empty($report->getName()) ? "Unknown" : $report->getName(), Breadcrumb::ERROR_TYPE, $report->getSummary());
Hope that helps.

Thanks for the detailed report, @jasminhassan. This is something we should fix!

Hi @jasminhassan, I'm looking into this issue at the moment.  Can you give me any more information about the report that's missing a name?  Can you find which factory method is creating it (i.e. fromPHPError, fromPHPThrowable, fromNamedError)?

Hi @Cawllec, Would a stack trace from OctoberCMS (based on Laravel 5.5) log be useful?
[2017-11-24 07:31:14] production.ERROR: InvalidArgumentException: The breadcrumb name must be a non-empty string. in /var/www/html/services/internal-october/vendor/bugsnag/bugsnag/src/Breadcrumbs/Breadcrumb.php:121
Stack trace:
#0 /var/www/html/services/internal-october/vendor/bugsnag/bugsnag/src/Client.php(214): Bugsnag\Breadcrumbs\Breadcrumb->_
_construct('', 'error', Array)
#1 /var/www/html/services/internal-october/vendor/bugsnag/bugsnag/src/Client.php(294): Bugsnag\Client->leaveBreadcrumb('', 'error', Array)
#2 /var/www/html/services/internal-october/vendor/bugsnag/bugsnag/src/Client.php(255): Bugsnag\Client->notify(Object(Bugsnag\Report), Object(Closure))
#3 /var/www/html/services/internal-october/vendor/mead-steve/mono-snag/src/BugsnagHandler.php(65): Bugsnag\Client->notifyError('', '[2017-11-24 07:...', Object(Closure))
#4 /var/www/html/services/internal-october/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(37): MeadSteve\MonoSnag\BugsnagHandler->write(Array)
#5 /var/www/html/services/internal-october/vendor/monolog/monolog/src/Monolog/Logger.php(337): Monolog\Handler\AbstractProcessingHandler->handle(Array)
#6 /var/www/html/services/internal-october/vendor/monolog/monolog/src/Monolog/Logger.php(616): Monolog\Logger->addRecord(400, NULL, Array)
#7 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Log/Writer.php(203): Monolog\Logger->error(NULL, Array)
#8 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Log/Writer.php(114): Illuminate\Log\Writer->writeLog('error', NULL, Array)
#9 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(221): Illuminate\Log\Writer->error(NULL)
#10 /var/www/html/services/internal-october/plugins/myorganization/dbtransfer/controllers/LoadSets.php(56): Illuminate\Support\Facades\Facade::__callStatic('error', Array)
#11 [internal function]: myorganization\DBTransfer\Controllers\LoadSets->onLoad('7')
#12 /var/www/html/services/internal-october/modules/backend/classes/Controller.php(536): call_user_func_array(Array, Array)
#13 /var/www/html/services/internal-october/modules/backend/classes/Controller.php(419): Backend\Classes\Controller->runAjaxHandler('onLoad')
#14 /var/www/html/services/internal-october/modules/backend/classes/Controller.php(229): Backend\Classes\Controller->execAjaxHandlers()
#15 /var/www/html/services/internal-october/modules/backend/classes/BackendController.php(112): Backend\Classes\Controller->run('dumps', Array)
#16 [internal function]: Backend\Classes\BackendController->run('myorganization...')
#17 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): call_user_func_array(Array, Array)
#18 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller->callAction('run', Array)
#19 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Route.php(212): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route), Object(Backend\Classes\BackendController), 'run')
#20 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Route.php(169): Illuminate\Routing\Route->runController()
#21 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Router.php(658): Illuminate\Routing\Route->run()
#22 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(30): Illuminate\Routing\Router->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#23 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(41): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#24 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\Routing\Middleware\SubstituteBindings->handle(Object(Illuminate\Http\Request), Object(Closure))
#25 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#26 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#27 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\View\Middleware\ShareErrorsFromSession->handle(Object(Illuminate\Http\Request), Object(Closure))
#28 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#29 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(63): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#30 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\Session\Middleware\StartSession->handle(Object(Illuminate\Http\Request), Object(Closure))
#31 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#32 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#33 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle(Object(Illuminate\Http\Request), Object(Closure))
#34 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#35 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(59): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#36 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\Cookie\Middleware\EncryptCookies->handle(Object(Illuminate\Http\Request), Object(Closure))
#37 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#38 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#39 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Router.php(660): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#40 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Router.php(635): Illuminate\Routing\Router->runRouteWithinStack(Object(Illuminate\Routing\Route), Object(Illuminate\Http\Request))
#41 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Router.php(601): Illuminate\Routing\Router->runRoute(Object(Illuminate\Http\Request), Object(Illuminate\Routing\Route))
#42 /var/www/html/services/internal-october/vendor/october/rain/src/Router/CoreRouter.php(20): Illuminate\Routing\Router->dispatchToRoute(Object(Illuminate\Http\Request))
#43 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(176): October\Rain\Router\CoreRouter->dispatch(Object(Illuminate\Http\Request))
#44 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(30): Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http\{closure}(Object(Illuminate\Http\Request))
#45 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/CheckForMaintenanceMode.php(46): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#46 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode->handle(Object(Illuminate\Http\Request), Object(Closure))
#47 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#48 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#49 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(151): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#50 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(116): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(Object(Illuminate\Http\Request))
#51 /var/www/html/services/internal-october/index.php(43): Illuminate\Foundation\Http\Kernel->handle(Object(Illuminate\Http\Request))
#52 {main}

The bug is here:

#9 /var/www/html/services/internal-october/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(221): Illuminate\Log\Writer->error(NULL)
#10 /var/www/html/services/internal-october/plugins/myorganization/dbtransfer/controllers/LoadSets.php(56): Illuminate\Support\Facades\Facade::__callStatic('error', Array)

You are trying to write null to the logs. You must pass non-null arguments to the PSR logger.

We should probably skip recording such breadcrumbs? That said, this did uncover a bug in october that you wouldn't have otherwise found.

We should probably skip recording such breadcrumbs?

By that I mean, skip calling leaveBreadcrumb for such cases, rather than leaveBreadcrumb silently skipping.

In terms of the actual issue with the notifier, it currently allows you to set an empty string as a valid report name, which then throws an exception as empty strings are invalid for breadcrumb names.
I would propose either allowing empty strings as breadcrumb names (not very useful but doesn't lose any data), or dropping nameless breadcrumbs with a debug log message.

I would not recommend either of those recommendations I'm afraid, but instead do the variant I suggested:


guile-gnome: fix build by building gwrap with guile-2.0

Motivation for this change
Fixes the build for guile-gnome, which currently does not work: https://github.com/NixOS/nixpkgs/pull/29732#issuecomment-331711946
@taktoa I also went ahead and removed you from maintainers for gwrap since it seems to only be a guile related package. (There's nothing to 'maintain' anyway since if you look at the commit messages in this PR, you'll see that it seems as though it's not being developed anymore, but wanted to ping you just in case.)
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Wait do not merge, I just discovered that you can set the guile version directly in pkgs/top-level/all-packages.nix, so I'll update this PR to do that instead some time.

:point_up_2: Done

Fixed the nitpicks myself, thanks for your work on the guile stuff! If you are still at it guile-xcb is the last one needing to be fixed ;)

Thanks @globin and thanks for reviewing @orivej! I think for guile-xcb, it just hasn't been updated to work with guile 2.2, so I made a PR for fixing that: #29916. But I'm not 100% sure because this line mentions 2.2: https://github.com/mwitmer/guile-xcb/blob/db7d5a393cc37a56f66541b3f33938b40c6f35b3/configure.ac#L12


Update gitignore

Signed-off-by: absarrahman<EMAIL_ADDRESS>

Hi,
Thanks for the PR. But what is this PR for?
Thanks

Cause if someone clones your repo, IDE related dependencies also gets
merged which breaks the VCS of the project. That's why I added it.
On Fri, Dec 25, 2020, 11:05 PM Amol Gangadhare<EMAIL_ADDRESS>wrote:

Hi,
Thanks for the PR. But what is this PR for?
Thanks
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/AmolGangadhare/flutter_barcode_scanner/pull/141#issuecomment-751272174,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIPM5UXEJGFX6M4RNSPRTT3SWTA6LANCNFSM4VFEYM7Q
.

Looks like you also removed funding.yml and other files in .github folder

Looks like you also removed funding.yml and other files in .github folder

Looks like you also removed funding.yml and other files in .github folder

Cause it will create conflicts with other people. And .github folder automatically sets based on the user. Everyone has different .github folder.

Looks like you also removed funding.yml and other files in .github folder

Cause it will create conflicts with other people. And .github folder automatically sets based on the user. Everyone has different .github folder.

Looks like you also removed funding.yml and other files in .github folder

Cause it will create conflicts with other people after cloning. And .github folder automatically sets based on the user. Everyone has different .github folder.

Wow just wow, looks like you are having incorrect info or misconfusion b/w .github folder as it doesnt have different files for every other clones or forks.
If you look closely what you have deleted, you will find you have deleted Funding.yml which is used for sponsoring the project, reference below:

And you have deleted bug report and feature report md files which are used as a base template when creating a new issue, reference below:

I am not judging you or have any personal war but I am only saying that you should not have .github folder.

Looks like you also removed funding.yml and other files in .github folder

Cause it will create conflicts with other people after cloning. And .github folder automatically sets based on the user. Everyone has different .github folder.

Wow just wow, looks like you are having incorrect info or misconfusion b/w .github folder as it doesnt have different files for every other clones or forks.
If you look closely what you have deleted, you will find you have deleted Funding.yml which is used for sponsoring the project, reference below:

And you have deleted bug report and feature report md files which are used as a base template when creating a new issue, reference below:

I am not judging you or have any personal war but I am only saying that you should not have .github folder.

I am so sorry :( Forgive me for this mistake. I really did not want to
cause any inconvenience.
On Sun, Jan 17, 2021, 10:18 AM Prateek SU<EMAIL_ADDRESS>wrote:

@prateekmedia requested changes on this pull request.
Dont remove .github folder files
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/AmolGangadhare/flutter_barcode_scanner/pull/141#pullrequestreview-569985499,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIPM5UVAW6FBLSJ7FJPAPJDS2JQH5ANCNFSM4VFEYM7Q
.

I am so sorry :( Forgive me for this mistake. I really did not want to
cause any inconvenience.
On Sun, Jan 17, 2021, 10:18 AM Prateek SU<EMAIL_ADDRESS>wrote:

@prateekmedia requested changes on this pull request.
Dont remove .github folder files
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/AmolGangadhare/flutter_barcode_scanner/pull/141#pullrequestreview-569985499,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIPM5UVAW6FBLSJ7FJPAPJDS2JQH5ANCNFSM4VFEYM7Q
.


its importation fixes and tests

Description
Progress

[ ] tests
[ ] documentation
[ ] implementation

Associated issues
#137

For the record, this is the R code that was used to generate the truth CSV files for the tests:
$ cat generate_truth.R
ITSbin::ITS_to_seconds(
  "examples/valid_raw_data/annotations/new_its/raw",
  "tests/truth",
  write.recordings = FALSE,
  write.blocks = FALSE,
  write.segments = TRUE,
  write.centiseconds = FALSE,
  write.seconds = FALSE,
  time.zone = "Europe/Amsterdam"
)

ITSbin::ITS_to_seconds(
  "examples/valid_raw_data/annotations/old_its/raw",
  "tests/truth",
  write.recordings = FALSE,
  write.blocks = FALSE,
  write.segments = TRUE,
  write.centiseconds = FALSE,
  write.seconds = FALSE,
  time.zone = "Europe/Amsterdam"
)


format("z") should be 0-based

PHP uses zero-based numbering for "z" in date formatting and returns values from 0 - 365. This library returns values from 1 - 366.
In my opinion, to fully mimic PHP, the function should end with a subtraction of 1:
(linebreaks added for readability)
z: function () {
  var d = new Date(this.getFullYear(), 0, 1);
  return Math.ceil((this - d) / 86400000) - 1
},

Or was there a motivation to not used zero-based numbering here?

Hello,
If you to be zero based you can use Math.floor instead of math.ceil and remove -1 :)
var now = new Date();
var start = new Date(now.getFullYear(), 0, 1);
var d = new Date(now.getFullYear(), 0, 1);
alert(Math.floor((now - d) / 86400000));
I hope that will resolve your issue .
Br,
Elhidery

@elhidery unless I'm mistaken the question not about how to convert the function to be zero based (which is trivial indeed ;-). It is about  (quoting the question): was there a motivation to not used zero-based numbering here?

Hello again,
I apologize for or misunderstanding.
This below line of code
var d = new Date(this.getFullYear(), 0, 1);
This.getFullYearr returns the current year which is "2018"  the "0" refers to january because the months are zero based ,finally the day is stored as 1 because the domain in  javascript is formatted from 1 to 31 ,if the day is written as "0" it will translate to the final day of the corresponding month.
So the modulation to not use zero based numbering is to avoid conflicts with the javascript standards in which the value of "0" for the day corresponds to the maximum day value of the corresponding month.

@elhidery unless I'm mistaken the question is not about the rationale for how zero is used in the Date function argument. It is about the range of values returned by the z function copy/pasted in the message.

The documentation in the README should probably be updated, then? It says "0-365" ;)


[Bug] Build failure due to expired signing certs

Archive building in latest main branch build is failing due to the expiry of signing cert on 13 Feb.

Adopt guidance here: https://docs.github.com/en/actions/deployment/deploying-xcode-applications/installing-an-apple-certificate-on-macos-runners-for-xcode-development


feat: implement ls-refs upload-pack command

Implement ls-refs command handling of git-upload-pack service:

implement command parsing of git client.
add LsRefsSlice to handle ls-refs command of git-upload-pack
service.
add TmpResource to copy git repository from storage to temp
directory.
implement command routing for upload-pack service.
enable and fix integration test for ls-refs command.

@olenagerasimova please check all tests passes


Write ballerina test cases for HTTP/2 body compression and 100-continue scenarios

$subject
Tests needed for the following (both prior on and prior off)

[x] body compression
[x] 100 continue server side handling
[x] unexpected 100-continue response handling in client side

We cannot add e2e tests for compression as the body is anyway decompressed when coming to the users end.
Therefore, tested the body compression scenario with nghttp client.
Following are the logs.
Without compression
[  0.028] recv (stream_id=13) :status: 200
[  0.028] recv (stream_id=13) content-type: text/plain
[  0.028] recv (stream_id=13) content-encoding: identity
[  0.028] recv (stream_id=13) server: ballerina
[  0.028] recv (stream_id=13) date: Wed, 19 Apr 2023 23:14:46 +0530
[  0.028] recv HEADERS frame <length=67, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0)
          ; First response header
This is a very very very very very very very very long string for compression testing.[  0.029] recv DATA frame <length=86, flags=0x01, stream_id=13>
          ; END_STREAM
[  0.029] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

With compression
[  0.025] recv (stream_id=13) :status: 200
[  0.025] recv (stream_id=13) content-type: text/plain
[  0.025] recv (stream_id=13) server: ballerina
[  0.025] recv (stream_id=13) date: Wed, 19 Apr 2023 23:15:26 +0530
[  0.025] recv (stream_id=13) content-encoding: gzip
[  0.025] recv HEADERS frame <length=63, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0)
          ; First response header
This is a very very very very very very very very long string for compression testing.[  0.029] recv DATA frame <length=74, flags=0x01, stream_id=13>
          ; END_STREAM
[  0.029] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

Tested for a very large string as well. The uncompressed size was 2408 bytes and the compressed size is 96 bytes.
Closing this issue as all the tasks are done.


Update tutorial-create-and-manage-powershell.md

Fix a VM instance size.

@alexandair : Thanks for your contribution! The author, @iainfoulds, has been notified to review your proposed change.


[Material3] IconButton's isSelected not visible to screen readers

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce

Run my sample, pasted below.
Tap the button in the center a few times. It has two states: selected and unselected.
Activate a screen reader (I used VoiceOver on iOS)
Use the screen reader to cause the button to be "pressed". (With VoiceOver, I found this is done with a double-tap.)
Notice that its tooltip ("hello") is read, but the selected/unselected state is not.

Expected results
I expected the selected/unselected state to be read aloud.
Actual results
The selected/unselected state was not read aloud.
Code sample
Code sample
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      theme: Theme.of(context).copyWith(useMaterial3: true),
      home: const Scaffold(
        body: Center(
          child: Column(
            mainAxisAlignment: MainAxisAlignment.center,
            children: [
              Text('Button:'),
              MyButton(),
            ],
          ),
        ),
      ),
    );
  }
}

class MyButton extends StatefulWidget {
  const MyButton({Key? key}) : super(key: key);

  @override
  State<MyButton> createState() => _MyButtonState();
}

class _MyButtonState extends State<MyButton> {
  bool _selected = false;

  @override
  Widget build(BuildContext context) {
    return IconButton(
      onPressed: () {
        setState(() {
          _selected = !_selected;
        });
      },
      tooltip: 'hello',
      isSelected: _selected,
      icon: const Icon(Icons.arrow_left),
      selectedIcon: const Icon(Icons.arrow_right),
    );
  }
}

Screenshots or Video

Screenshots / Video demonstration
N/A

Logs
Logs
No errors logged or expected.

Flutter Doctor output
Doctor output
$ flutter doctor -v
[✓] Flutter (Channel main, 3.11.0-6.0.pre.35, on macOS 13.3.1 22E772610a darwin-x64, locale en-US)
    • Flutter version 3.11.0-6.0.pre.35 on channel main at /Users/chrisbobbe/.local/lib/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 37fe1e9d77 (7 days ago), 2023-05-11 12:43:40 -0400
    • Engine revision f38f46f66e
    • Dart version 3.1.0 (build 3.1.0-94.0.dev)
    • DevTools version 2.23.1

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/chrisbobbe/Library/Android/sdk
    • Platform android-33, build-tools 30.0.3
    • ANDROID_HOME = /Users/chrisbobbe/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] VS Code (version 1.78.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (3 available)
    • Chris’s iPhone (mobile) • 00008110-000924383632801E • ios            • iOS 16.4.1 20E252
    • macOS (desktop)         • macos                     • darwin-x64     • macOS 13.3.1 22E772610a darwin-x64
    • Chrome (web)            • chrome                    • web-javascript • Google Chrome 113.0.5672.126
    ! Error: Chris’s Apple Watch needs to connect to determine its availability. Check the connection between the
      device and its companion iPhone, and the connection between the iPhone and Xcode. Both devices may also need to
      be restarted and unlocked. (code 1)

[✓] Network resources
    • All expected network resources are available.

• No issues found!

The following simple change does make the state visible to screen readers:
diff --git packages/flutter/lib/src/material/icon_button.dart packages/flutter/lib/src/material/icon_button.dart
index 7acd75762..2281d3318 100644
--- packages/flutter/lib/src/material/icon_button.dart
+++ packages/flutter/lib/src/material/icon_button.dart
@@ -874,7 +874,10 @@ class _SelectableIconButtonState extends State<_SelectableIconButton> {
       onPressed: widget.onPressed,
       variant: widget.variant,
       toggleable: toggleable,
-      child: widget.child,
+      child: Semantics(
+        checked: widget.isSelected,
+        child: widget.child),
     );
   }
 }

Ideally, though, IconButton would let its callers choose whether to represent the state with SemanticsProperties.checked or SemanticsProperties.toggled. I think callers could reasonably choose either under the Material 3 spec.

Linking https://github.com/flutter/flutter/pull/106437 for context; that's when the isSelected param was added.

Thanks for the report. Reproduced the issue on the latest stable and master channels.
Verified on VoiceOver (iOS) and TalkBack (Android).

flutter doctor -v (stable and master)
[✓] Flutter (Channel stable, 3.10.1, on macOS 13.0.1 22A400 darwin-x64, locale en-VN)
    • Flutter version 3.10.1 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision d3d8effc68 (26 hours ago), 2023-05-16 17:59:05 -0700
    • Engine revision b4fb11214d
    • Dart version 3.0.1
    • DevTools version 2.23.1

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 32.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.78.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.64.0

[✓] Connected device (3 available)
    • RMX2001 (mobile) • EUYTFEUSQSRGDA6D • android-arm64  • Android 11 (API 30)
    • macOS (desktop)  • macos            • darwin-x64     • macOS 13.0.1 22A400 darwin-x64
    • Chrome (web)     • chrome           • web-javascript • Google Chrome 113.0.5672.126

[✓] Network resources
    • All expected network resources are available.

• No issues found!

[!] Flutter (Channel master, 3.11.0-6.0.pre.153, on macOS 13.0.1 22A400 darwin-x64, locale en-VN)
    • Flutter version 3.11.0-6.0.pre.153 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision e49d35d3a6 (35 minutes ago), 2023-05-18 22:26:29 -0400
    • Engine revision bca11a423f
    • Dart version 3.1.0 (build 3.1.0-125.0.dev)
    • DevTools version 2.23.1
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 32.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.78.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.64.0

[✓] Connected device (3 available)
    • RMX2001 (mobile) • EUYTFEUSQSRGDA6D • android-arm64  • Android 11 (API 30)
    • macOS (desktop)  • macos            • darwin-x64     • macOS 13.0.1 22A400 darwin-x64
    • Chrome (web)     • chrome           • web-javascript • Google Chrome 113.0.5672.126

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.


feat: use top-level await to import server envs

Related to #603
✅ Checklist

[x] I have followed every step in the contributing guide (updated 2022-10-06).
[x] The PR title follows the convention we established conventional-commit
[x] I performed a functional test on my final commit

Changelog

Use top-level await in the next.config.mjs.
Change tsconfig target to es2017.

💯

Need:

changeset (so we can trigger a release)
mentioning in the env-vars docs on a) what it does and b) how to use it


Rejecting/Resolving aff multiple times in makeAff

Hey,
I apologise if this is not a bug but rather a hole in my understanding, but I was very surprised to see that when I make new aff's with makeAff/makeAff', I can resolve and reject them multiple times. Is this expected behavior? It feels unintuitive to me because it's the opposite of how eg. JavaScript's promises work.

For the pre-v4 release it's expected, as Aff supported continuation behaviour.

Great, thanks for the swift answer! Just out of curiosity, will this change in V4?

Yeah, in v4 the contination behaviour has been dropped, because it's confusing/surprising, as you found :smile:. It'd been intended to be deprecated for a while, but as there are other fairly major changes with v4 it seemed as good a time as any.

Awesome, thanks for the help!


Internal classes should not be part of PhpStormStubsMap

Thanks for the PR. Theses classes are required for our engine, they were added synthetically as a result of https://youtrack.jetbrains.com/issue/WI-60439 & https://youtrack.jetbrains.com/issue/WI-60460.


[ethernet-view] Name adjustments

While working with the TabLayout I realised that some names could be better, so this is a small PR with changes related to that. Nothing so important, just naming things

I like this PR, slowly making the code better 💪


#382 Vue warn: Avoid using non-primitive value as key, use string/number value instead.

Please review this issue https://github.com/ghosh/uiGradients/issues/382 and then these code.
Thanks.
Screenshot:
npm run dev, Browser console

Visual Studio Code, ESlint

Thank you. This has been fixed in another commit


NuGet task doesn't correctly error when "Path to NuGet.config" incorrect

When "Path to NuGet.config" is incorrectly set to a folder path instead of the NuGet.config path, instead of erroring, sometimes the task will continue, creating a temporary nuget.config with just "" in it. The build proceeds and then users are confused when their packages fail to restore because the sources/creds aren't there.
The expected behavior is that it would fail with a "you provided a directory, provide the exact path to nuget.config" type error. This does kinda happen in some cases ("illegal operation on a directory, read"), but not always. Also the current error message could obviously be improved.
I believe the difference in behavior (continuing with empty nuget.config, versus failing) seems to depend on if the folder provided is within the repository or not. Looking at the code, it seems like tl.filePathSupplied or our use of it is likely the culprit.

@satbai @keithrob

Hello! I'm a PM over on the Azure Artifacts team. I wanted to drop in here and address some of these issues with the news that we're currently reevaluating the types of tasks we offer in Azure Pipelines, and looking to focus more on lighter-weight tasks that cover authentication and setup. In lieu of the task redesign, we're going to forego making changes to the current tasks (with a few exceptions). We hope the new tasks we spin up in the next quarter or so are easier and more helpful, be on the lookout!
Adding: closing this issue because this looks like an inconvenience error instead of a bug that's blocking users.


Display approximate distance from user

Not sure exactly where, but it'd be a nice addition to display somewhere.
See here: http://stackoverflow.com/questions/10879579/how-to-calculate-distance-from-user-location-to-annotation-when-user-moves

Also closed by 73001dc30385769a58402badbd4cf735db5a4b77


[Manage Docs] Remove category details in case document is marked as confidential

Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)
[ ] the keep-helm label has been added, if the helm release should be persisted after a successful build

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
Change description
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

Keeping category details as selected during upload, it will display details even if document is moved to confidential.
So closing this PR.


Remove UseGPU in GBM.

Clean up after removing n_gpus.

Let's don't rush into this for now.  The dependent on GPUSet is quite risky.


Please can someone help me connect to HA Tesla Powerwall app (not Custom)

On my mobile phone I have logged into the TEG-XXX network, although cannot log into a browser once connected. Once the WiFi is connected I then looked up the IP Address shown, I then used this IP when asked trying to get into the HA app followed by the password which I use the last 5 characters of my Powerwall Serial Number, but it fails every time. Any help much appreciated.

This is not the right place to ask this.


[feat] Add system uptime to Metrics page

Hours I've been alive, not "since when". Return in server metrics response @dr-bonez (commented moved from incorrect ticket)

@Blu-J should be the first item under "General" in the server metrics response


Issue with converting Haystack filters to Qdrant style

Describe the bug
I'm using the Qdrant document store integration with Haystack. I'm running a retrieval pipeline where I'd like to filter search results using metadata. I have created the filter in Haystack dictionary format, and this gets converted to Qdrant filter format when I run the pipeline and provide the filter dictionary to the retriever components.
The problem is the filter dictionary I'm using, which is a little complex, doesn't get converted properly to Qdrant format, some of the filters are completely absent from the Qdrant version.
To Reproduce
Here is an example of the type of Haystack filter dictionary I'd like to convert:
filter = {
    "operator": "AND",
    "conditions": [
        {
            "field": "meta.file_name",
            "operator": "in",
            "value": [
                "file1",
                "file2"
            ]
        },
        {
            "operator": "OR",
            "conditions": [
                {
                    "field": "\"meta\".\"classification\".\"details\".\"category1\"",
                    "operator": ">=",
                    "value": 0.85
                },
                {
                    "field": "\"meta\".\"classification\".\"details\".\"category2\"",
                    "operator": ">=",
                    "value": 0.85
                }
            ]
        }
    ]
}

If I convert this using convert_filters_to_qdrant the output is:
from haystack_integrations.document_stores.qdrant.filters import convert_filters_to_qdrant

qdrant_filter = convert_filters_to_qdrant(filter)
qdrant_filter =
    "must": [
        {
            "should": [
                {
                    "key": "meta.file_name",
                    "match": {
                        "text": "file1"
                    }
                },
                {
                    "key": "meta.file_name",
                    "match": {
                        "text": "file2"
                    }
                }
            ]
        }
    ]
}

Here, the category1 and category2 filters are missing from the output, so my search results don't get filtered properly.
What I'd expect to see is the category filters to be present in the qdrant version output by convert_filters_to_qdrant and in a way that reflects the logic correctly in my original Haystack filter.
Describe your environment

OS: macOS Version 14.5
Haystack version: 2.1.0
Integration version: qdrant-haystack version 3.5.0

This indeed a bug in the conversion, though I'm wondering why are you writing the field like that? 🤔
meta.classification.details.category1 this seems more correct to me. It doesn't fix the bug though.

This indeed a bug in the conversion, though I'm wondering why are you writing the field like that?

I wrote it this way to allow for categories containing multiple words with spaces. E.g. if category1 was actually "popular science" using that formatting the filtering works. Comes from this response to an Issue I raised.


custom cloudfront configurations

Solves #270. Allows configuration for both default and api lambda cloudfront options. It will silently ignore any setting to "lambda@edge" (since this should never be changed). Also, besides default and api, no other cloudfront setting will be honored.

Cheers for the PR Barry. Overall LGTM 🚀 Just a few comments as above.


Can't import google.protobuf.timestamp_pb2 from App Engine

I'm getting ImportError: No module named protobuf from the line from google.protobuf import timestamp_pb2 because the google module refers to the AppEngine SDK.
This is a well-known issue that's been around for years. Maybe you can bundle protobuf directly inside gcloud-python instead of an external dependency?

Thanks for reporting and for upgrading!
I am curious about a few things:

How do you vendor in gcloud-python for App Engine?
Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)
Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)
 If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

 If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

I don't think this shouldn't be necessary anymore, I'm interested to see if that's somehow changed.

Me too. Thanks for chiming in @jonparrott I was going to ping you.

I should really write an App Engine test case.

@SaMnCo also reported this import error, it may be

we don't understand our requirements like we think we do.
pip doesn't upgrade deps in setup.py as I expect
a PYTHONPATH issue

@SaMnCo can you do a pip show protobuf, pip show gcloud in the environment where this is failing and then make sure the same versions are imported:
$ python -c 'import google.protobuf; print(google.protobuf.__version__)'
$ python -c 'import gcloud; print(gcloud.__version__)'

@dhermes  I think I've got a similar problem as the reporter of the issue. Perhaps I can help by providing my answers to your questions:

How do you vendor in gcloud-python for App Engine?

In appengine_config.py:
from google.appengine.ext import vendor
vendor.add('libs')
import google.protobuf; print(google.protobuf.__version__)
import gcloud; print(gcloud.__version__)

Results in:
3.0.0b2
0.10.0

Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)

It's in libs, so yes.

Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)

  from gcloud import storage
File "/x/libs/gcloud/storage/__init__.py", line 43, in <module>
  from gcloud.storage.blob import Blob
File "/x/libs/gcloud/storage/blob.py", line 27, in <module>
  from gcloud._helpers import _rfc3339_to_datetime
File "/x/libs/gcloud/_helpers.py", line 26, in <module>
  from google.protobuf import timestamp_pb2
ImportError: No module named protobuf

@kvdb Thanks a lot. I'm wondering now if it may be related to Darth-Vendor and namespace packages not being the best of friends. I need to dig in and reproduce. Can you confirm that's what you used to vendor in your package?

Darth is part of the SDK (from google.appengine.ext import vendor) so that's definitely what's being used. It explicitly has support for namespaces. So I'm also unsure what's going on.

Maybe it's an issue with old versions of pip? The module certainly shows up:
$ mkdir lib
$ pip install -t lib gcloud
$ cd lib/google/protobuf/
$ ls -1 *pb2.py | egrep -v unittest
any_pb2.py
api_pb2.py
descriptor_pb2.py
duration_pb2.py
empty_pb2.py
field_mask_pb2.py
source_context_pb2.py
struct_pb2.py
timestamp_pb2.py
type_pb2.py
wrappers_pb2.py

Going to run a test app in dev_appserver to verify import works.

OK I just confirmed this works in dev_appserver as well.
@kvdb Have you tried clearing out lib and starting fresh? What version of pip are you using?

pip 8.0.2
Yes, I cleared my virtualenv/libs before reporting.

Very strange! Does the timestamp_pb2.py file exist and just the import fails or the file never shows up?

I was having this error outside App Engine when using an old version of pip and not using the full command to install:
pip install --upgrade gcloud

I was using pip -r requirements.txt instead of pip --upgrade -r requirements.txt and the timestamp_pb2.py file was not present.

Outside App Engine means inside Docker, using the google/debian:wheezy image.

Thanks for the report @ernestoalejo! @kvdb any thoughts?

@kvdb Does the timestamp_pb2.py file exist and just the import fails or the file never shows up?

I'm closing for now since I can't reproduce and @ernestoalejo answer indicates --upgrade resolves the issue. @kvdb I'm happy to re-open if it still comes up.

@vsubramani can you give some info on your install:
from google import protobuf
print(protobuf.__version__)
print(protobuf.__path__)
print(protobuf.__path__[0])
import os
pb_files = sorted([fi for fi in os.listdir(protobuf.__path__[0])
                   if fi.endswith('.py')])
print('\n'.join(pb_files))

So I had the same issue running from gcloud import storage inside a docker container output an error.
In [3]: from gcloud import storage
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-a2d587421817> in <module>()
----> 1 from gcloud import storage

/venv/lib/python2.7/site-packages/gcloud/storage/__init__.py in <module>()
     41
     42 from gcloud.storage.batch import Batch
---> 43 from gcloud.storage.blob import Blob
     44 from gcloud.storage.bucket import Bucket
     45 from gcloud.storage.client import Client

/venv/lib/python2.7/site-packages/gcloud/storage/blob.py in <module>()
     25 from six.moves.urllib.parse import quote
     26
---> 27 from gcloud._helpers import _rfc3339_to_datetime
     28 from gcloud.credentials import generate_signed_url
     29 from gcloud.exceptions import NotFound

/venv/lib/python2.7/site-packages/gcloud/_helpers.py in <module>()
     25 from threading import local as Local
     26
---> 27 from google.protobuf import timestamp_pb2
     28 import six
     29 from six.moves.http_client import HTTPConnection

ImportError: cannot import name timestamp_pb2

Running
>>> from google import protobuf
>>> print(protobuf.__version__)
>>> print(protobuf.__path__)
>>> print(protobuf.__path__[0])
>>> import os
>>> pb_files = sorted([fi for fi in os.listdir(protobuf.__path__[0])
                   if fi.endswith('.py')])
>>> print('\n'.join(pb_files))
__init__.py
descriptor.py
descriptor_database.py
descriptor_pb2.py
descriptor_pool.py
message.py
message_factory.py
proto_builder.py
reflection.py
service.py
service_reflection.py
symbol_database.py
text_encoding.py
text_format.py

This was fixed by adding this line into my Dockerfile
RUN pip install --upgrade gcloud

It looks like you have to force the gcloud installation to upgrade. This was the output while building my docker container.
Step 13 : RUN pip install --upgrade gcloud
 ---> Running in 4670a504231c
Requirement already up-to-date: gcloud in /venv/lib/python2.7/site-packages
Requirement already up-to-date: pyOpenSSL in /venv/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: six in /venv/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: httplib2>=0.9.1 in /venv/lib/python2.7/site-packages (from gcloud)
Collecting protobuf!=3.0.0.b2.post1,>=3.0.0b2 (from gcloud)
  Downloading protobuf-3.0.0b2.post2-py2-none-any.whl (331kB)
Collecting oauth2client>=2.0.1 (from gcloud)
  Downloading oauth2client-2.0.2.tar.gz (68kB)
Requirement already up-to-date: googleapis-common-protos in /venv/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: cryptography>=1.3 in /venv/lib/python2.7/site-packages (from pyOpenSSL->gcloud)
Requirement already up-to-date: setuptools in /venv/lib/python2.7/site-packages (from protobuf!=3.0.0.b2.post1,>=3.0.0b2->gcloud)
Requirement already up-to-date: pyasn1>=0.1.7 in /venv/lib/python2.7/site-packages (from oauth2client>=2.0.1->gcloud)
Requirement already up-to-date: pyasn1-modules>=0.0.5 in /venv/lib/python2.7/site-packages (from oauth2client>=2.0.1->gcloud)
Collecting rsa>=3.1.4 (from oauth2client>=2.0.1->gcloud)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
Collecting enum34 (from cryptography>=1.3->pyOpenSSL->gcloud)
  Downloading enum34-1.1.3-py2.py3-none-any.whl (61kB)
Requirement already up-to-date: ipaddress in /venv/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->gcloud)
Requirement already up-to-date: idna>=2.0 in /venv/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->gcloud)
Requirement already up-to-date: cffi>=1.4.1 in /venv/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->gcloud)
Requirement already up-to-date: pycparser in /venv/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->gcloud)
Building wheels for collected packages: oauth2client
  Running setup.py bdist_wheel for oauth2client: started
  Running setup.py bdist_wheel for oauth2client: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/3f/1b/4b/7d18513fd8ed10976cf4bb881bd0a2b549d4d22d245d0ff1a6
Successfully built oauth2client
Installing collected packages: protobuf, rsa, oauth2client, enum34
  Found existing installation: protobuf 3.0.0a3
    Uninstalling protobuf-3.0.0a3:
      Successfully uninstalled protobuf-3.0.0a3
  Found existing installation: rsa 3.3
    Uninstalling rsa-3.3:
      Successfully uninstalled rsa-3.3
  Found existing installation: oauth2client 1.5.2
    Uninstalling oauth2client-1.5.2:
      Successfully uninstalled oauth2client-1.5.2
  Found existing installation: enum34 1.1.2
    Uninstalling enum34-1.1.2:
      Successfully uninstalled enum34-1.1.2
Successfully installed enum34-1.1.3 oauth2client-2.0.2 protobuf-3.0.0b2.post2 rsa-3.4.2

Thanks for the note. Unfortunately there's not much we can do about the way Python packaging works beyond declaring the minimum versions we require.

The problems are still present using the latest gcloud.
In setup.py, I read the protobuf dependencies are defined as:
'protobuf >= 3.0.0b2, != 3.0.0.b2.post1'

I think I had problematic version 3.0.0b2 cached, that's why it was continued to be used within gcloud.
When upgrading protobuf to protobuf-3.0.0b2.post2, it works.
Therefore, I'd recommend defining the dependency in setup.py as follows:
'protobuf >= 3.0.0b2.post2'

What do you mean by a "problematic version 3.0.0b2"? I just did a fresh install and timestamp_pb2 is present:
$ virtualenv venv
$ source venv/bin/activate
(venv) $ pip install 'protobuf == 3.0.0b2.post2'
(venv) $ pip install 'protobuf == 3.0.0b2.post2'
(venv) $ ls venv/lib/python2.7/site-packages/google/protobuf/timestamp* -1
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.py
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.pyc

Ahhh hold on:
(venv) $ pip show protobuf
---
Metadata-Version: 2.0
Name: protobuf
Version: 3.0.0b2.post2
...

OK I'll use the correct version this time
$ virtualenv venv
$ source venv/bin/activate
(venv) $ pip install 'protobuf == 3.0.0b2'
(venv) $ ls venv/lib/python2.7/site-packages/google/protobuf/timestamp* -1
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.py
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.pyc
(venv) $ pip show protobuf
---
Metadata-Version: 2.0
Name: protobuf
Version: 3.0.0b2
...

I see your point, if I can reproduce it properly, I'll get back to you.

Thanks!

I'm having this exact same problem.....it's due to the order of entries in sys.path.......
here is my (abbreviated) path:
~/dev/NEW_TouchstoneMicroservices
/usr/local/google_appengine
~/dev/NEW_TouchstoneMicroservices/venv/lib/python27.zip
~/dev/NEW_TouchstoneMicroservices/venv/lib/python2.7
...
/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7
....
~/dev/NEW_TouchstoneMicroservices/venv/lib/python2.7/site-packages
You can see that:
/usr/local/google_appengine
comes BEFORE
..../venv/lib/python2.7/site-packages
and there is a "google" directory in both locations...
but the Google directory in "google_appengine" does not contain the "protobuf" module....
Unfortunately, because I'm running the endpoints (python /usr/local/bin/endpointscfg.py get_discovery_doc -f rpc) script, I don't have the option to modify the python path in advance....
I love the Google PAAS but the dev-server config stuff is a constant frustration...
how do I fix this??

@jonparrott any recommendations? @dgaedcke normally I would just say modify your PYTHONPATH, but I'm not sure if the GAE appserver hijacks the path.

I don't think there's a good answer for endpointscfg.py, other than modifying it directly. I'll assign this to me to reproduce and try to suggest a temporary workaround.

@dgaedcke it would be great if you could give me as much information as possible about:

How you install your packages
What's in your appengine_config.py
Does dev_appserver.py work for you?

@jonparrott  -- thanks for your help
1)  SETUP

I installed Python 2.7 & pip with Homebrew  (OSX El Capitan)
I create a venv directory inside my project
I activate the venv and then
pip --upgrade -r requirements.txt
then I symlink "venv/lib/python2.7/site-packages" to "lib" at top level of my project dir

2)  appengine_config
import endpoints
# I had to add the next line because Ferris was throwing an error about not being able
# to find that key on the module
endpoints.API_EXPLORER_CLIENT_ID = '292824132082.apps.googleusercontent.com'

from google.appengine.ext import vendor
vendor.add('lib')

3)  dev_appserver.py
it works as long as I comment out any attempt to run:
from gcloud import storage
if I leave that line enabled, it get the import problem described in this thread...

ext.vendor should support virtualenvs directly. Can you try not symlinking and just doing vendor.add('venv')?
Is this an App Engine Standard application or a Flexible app using the compat runtime?

it's a standard app......I'm not even sure what Flexible is but I've seen it pop up lately in some docs...
sure, I can try that but as you saw in my initial post, the problem stems from both directories having a "google" directory, but the first one does not have a "protobuf" module;   I would not expect the symlink to change PATH order.....
back shortly with results...

It won't, but it does help reduce weirdness. Less weirdness is better.

got a bit farther this time.....
I don't have 100% test coverage, so I've learned that the best way to exercise ALL of my imports is to run "endpointscfg.py" and so that's how I generated this trace:
Traceback (most recent call last):   File "/usr/local/bin/endpointscfg.py", line 133, in <module>     run_file(__file__, globals())   File "/usr/local/bin/endpointscfg.py", line 129, in run_file     execfile(_PATHS.script_file(script_name), globals_)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 561, in <module>     main(sys.argv)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 557, in main     args.callback(args)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 443, in _GenDiscoveryDocCallback     application_path=args.application)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 302, in _GenDiscoveryDoc     application_path=application_path)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 187, in GenApiConfig     module = __import__(module_name, fromlist=base_service_class_name)   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 27, in <module>     from uidisplay_service import DisplayService   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/uidisplay_service.py", line 27, in <module>     from common.blobstore.storage import upload_file   File "/Users/dgaedcke/dev/TouchstoneMicroservices/common/blobstore/storage.py", line 18, in <module>     from gcloud import storage   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/gcloud/storage/__init__.py", line 45, in <module>     from gcloud.storage.client import Client   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/gcloud/storage/client.py", line 19, in <module>     from gcloud.client import JSONClient   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/gcloud/client.py", line 17, in <module>     from oauth2client.service_account import ServiceAccountCredentials ImportError: cannot import name ServiceAccountCredentials

I wasn't able to reproduce your timestamp_pb2 import. I did the following:

virtualenv venv
venv/bin/pip install gcloud (note: I did not activate the virtualenv, but it didn't seem to make a difference.
added vendor.add('venv') to dev_appserver.py.
imported gcloud.storage in my app.
ran dev_appserver.py.

Okay, the new error you have is more familiar to me. Let me track down the workaround for that.

The new error is likely just the wrong version of oauth2client (pre-2.0) which may also be a path error (since oauth2client ships with the SDK)

@dgaedcke see #1705. In summary:
Use an appengine_config.py like so:
"""
`appengine_config.py` is automatically loaded when Google App Engine
starts a new instance of your application. This runs before any
WSGI applications specified in app.yaml are loaded.
"""

import sys
from google.appengine.ext import vendor

# Third-party libraries are stored in "lib", vendoring will make
# sure that they are importable by the application.
vendor.add('lib')

def unload_module(module_name):
    target_modules = [m for m in sys.modules if m.startswith(module_name)]
    for m in target_modules:
        if m in sys.modules:
            del sys.modules[m]

unload_module('oauth2client')
unload_module('pyasn1_modules')

Make sure you have this at the top of your API-defining modules (e.g., the scripts referenced by app.yaml):
try:
    appengine_config
except NameError:
    import appengine_config

that fixed it!!   Thank you guys very much......
it seems like that also solved the problem of Ferris not being able to find:  API_EXPLORER_CLIENT_ID
question for you:
All of my "main" (API defining) modules already had:
import appengine_config
with no problems......so why is the try-except approach preferred

Just in case you don't have it. You don't have to use the try-except approach. The magic was in unloading the modules.

I'm assuming one of my venv packages pulled down these "unloaded" modules as dependencies.....
Will it create a problem if those packages re-import them again at some later point???

I'm not sure what you're asking, but the crux of the issue is this:

You start endpointscfg.py.
endpointscfg.py import oauth2client. At this point, sys.path only contains the App Engine SDK, so the App Engine SDK's version of oauth2client<2.0 is loaded into sys.path.
endpointscfg.py tries to import your code.
Your code imports appengine_config.py which does path manipulation.
Your code tries to import gcloud, which in turn tries to import oauth2client.
Because endpointscfg.py had previously import oauth2client, it's already in sys.modules so not search is done and the newer, vendored version isn't used.
Because gcloud expects oauth2client>2.0 but gets oauth2client<2.0 it explodes.

Adding the trickery to unload oauth2client in appengine_config.py allows things to work as expected.

that clarification helps a lot....thanks for explaining!!!
And you answered my question......any modules that (at some point later) try to import "oauth2client" will get the  newer (>2.0) version because that's what's cached in sys.modules
so as long as those non-google packages don't have problems utilizing the newer oauth2client version, I should be golden....

Also, it's really nice that:
ext.vendor supports virtualenvs directly
perhaps I missed it, but I don't think this is well published or widely understood....most of the tutorials out there suggest the symlink approach

most of the tutorials out there suggest the symlink approach

Most of ours should recommend using pip install -t, do you have any links to tutorials that recommend the symlink?

I don't think this is well published

This on purpose. ;)

your recommendation of pip install -t lib presumes that it always works.....not sure if it's still the case, but pip from Homebrew would not do this properly for awhile.  Furthermore, your own article says GAE should support virtualenv....
Here are a few other links:
Vendoring done right
so
more
Also, I spoke too soon about that Ferris issue.....that problem only surfaces when we hit the endpoint, not when running endpointscfg so I thought it was solved and its not:
Traceback (most recent call last):   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle     handler = _config_handle.add_wsgi_middleware(self._LoadHandler())     handler = _config_handle.add_wsgi_middleware(self._LoadHandler())     handler = _config_handle.add_wsgi_middleware(self._LoadHandler())   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler     handler, path, err = LoadObject(self._handler)   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler     handler, path, err = LoadObject(self._handler)     handler, path, err = LoadObject(self._handler)   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject     __import__(cumulative_path)     __import__(cumulative_path)   File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject     __import__(cumulative_path)   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 9, in <module>   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 9, in <module>   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 9, in <module>     import endpoint_config     # establishes the endpoint 'tstone' in registry as the default     import endpoint_config     # establishes the endpoint 'tstone' in registry as the default     import endpoint_config     # establishes the endpoint 'tstone' in registry as the default   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/endpoint_config.py", line 4, in <module>   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/endpoint_config.py", line 4, in <module>   File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/endpoint_config.py", line 4, in <module>     endpoints.add('service_backend/endpoint_config.yaml', default=True)     endpoints.add('service_backend/endpoint_config.yaml', default=True)     endpoints.add('service_backend/endpoint_config.yaml', default=True)   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 37, in add   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 37, in add   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 37, in add     config = load_config_file(config_or_file)     config = load_config_file(config_or_file)     config = load_config_file(config_or_file)   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 91, in load_config_file   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 91, in load_config_file   File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 91, in load_config_file     recursive_replace(config, 'API_EXPLORER_CLIENT_ID', endpoints.API_EXPLORER_CLIENT_ID) AttributeError: 'module' object has no attribute 'API_EXPLORER_CLIENT_ID'     recursive_replace(config, 'API_EXPLORER_CLIENT_ID', endpoints.API_EXPLORER_CLIENT_ID)     recursive_replace(config, 'API_EXPLORER_CLIENT_ID', endpoints.API_EXPLORER_CLIENT_ID) AttributeError: 'module' object has no attribute 'API_EXPLORER_CLIENT_ID'

My own blog posts say a lot of things, they don't all make it into App Engine. :)
(As an aside, I find it funny that a blog post titled "Vendoring done right" does it wrong (though I don't blame him, this is very difficult stuff). Symlinking is never the answer.)
For the ferris issue, can you file that over on the ferris repo? I can investigate there.

haha....even worse....before you told me the vendor.add('venv') was intentionally not promoted, I emailed him (done right guy) to let him know he could skip the symlink ;-)   this mis-information grows like a virus

It's not promoted simply due to the fact that the behavior of the virtualenv is completely different from how virtualenvs work elsewhere. We don't want to promote something that has surprising, and especially idiosyncratic, behavior.

Hey Jon.....I clearly misunderstood something.....so where are the official "how to vendor" instructions....
When you mentioned that you could vendor.add("venv") directly, and not to use symlinks, I did that....
But of course now everything runs fine on the dev-server, but on the Production VM, it can't find any of it's dependencies.....that should have been obvious to me but it wasn't until I tried it....
So I guess I'm back to "pip install -t lib" but then I think I still need the venv for automated testing outside of the dev-server environment.....this package management stuff is really causing me a lot of pain....

ok...the pain continues....
I tried:   pip install -r requirements.txt -t lib
and I get:
Exception: Traceback (most recent call last):   File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main     status = self.run(options, args)   File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run     prefix=options.prefix_path,   File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 732, in install     **kwargs   File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 835, in install     self.move_wheel_files(self.source_dir, root=root, prefix=prefix)   File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 1030, in move_wheel_files     isolated=self.isolated,   File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 247, in move_wheel_files     prefix=prefix,   File "/usr/local/lib/python2.7/site-packages/pip/locations.py", line 153, in distutils_scheme     i.finalize_options()   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py", line 264, in finalize_options     "must supply either home or prefix/exec-prefix -- not both" DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both
such a drag to go round & round with this as many times as I have.....

Here is another guy promoting and publishing the symlink approach.......his is the newest and most consolidated instructions I've found.
Meanwhile, the "vendor" link on Google's own page seems to be broken.....

User error (at least in part).....my "venv" folder was part of my skip_files config.....so I guess it would have been deployed to the Production VM if that line was not in there.....but it seem quite weird and inefficient to be loading the WHOLE venv directory rather than just the site-packages subfolder....
I think I'm going to just sit tight until someone points me to the valid official google instructions for local dev setup....

Okay, lots of stuff to respond to here.

my "venv" folder was part of my skip_files config

I suspected as such, glad you figured it out.

but it seem quite weird and inefficient to be loading the WHOLE venv directory rather than just the site-packages subfolder

This is one of many reasons we don't recommend using a virtualenv for vendoring. You can setup skip_files to only upload the site-packages folder.

I think I'm going to just sit tight until someone points me to the valid official google instructions for local dev setup....

The instructions are here. The vendoring link is broken but it is supposed to link to this heading in the same page.
In summary:

Add vendor.add('lib') to appengine_config.py.
Use pip install -t lib -r requirements.txt.

These are the steps used in all of our samples for App Engine standard that require third-party packages.
From what I understand, this is where things deviated for you:

You initially used a virtualenv and a symlinked lib folder, neither of which are documented or really recommended by us, for various reasons.
You had your virtualenv ignored in app.yaml.

From what I can gather, you chose this route because pip install -t failed for you. The error you pasted above seems to implicate this known issue with homebrew Python. They have a workaround in their documentation.
It seems my actions here to ameliorate this are:

Fix the vendor link in the docs.
Add a note about homebrew Python not working with pip install -t and link to their workaround.

I feel your pain here, I've been personally working on improving this situation. Please let me know if there's anything more I can do at this time to make this easier.

Hey Jon..... thanks so much for your diligent responses & help!
A few clarifications here:
You link to this Homebrew/Python post as the "workaround" to the pip -t lib problem....but it's not clear to me what that post is suggesting because it's related to --user, not -t lib......I'm sure someone with more experience would see the connection instantly, but how to apply this workaround to my issue is lost on me...
Even though you say I could "setup skip_files to only upload the site-packages folder."....you are specifically recommending AGAINST that approach......correct?
But you still think it's ok to have a VENV in parallel to the -t lib/ approach to use only for automated testing purposes?  Or is it better to forego that also and somehow force the tests to use the GAE Python and it's respective paths??  Essentially running all tests in the exact same environment as the local dev server??   If so, can you link to a tutorial on how to do that??

pip install -t is just another form of --user internally. Creating the ~/.distutils.cfg as described will fix the issue. You can remove it after vendoring is done.

you are specifically recommending AGAINST that approach......correct?

Yes, I do not recommend using virtualenvs for vendoring on app engine.

But you still think it's ok to have a VENV in parallel to the -t lib/ approach to use only for automated testing purposes?

I'm not sure what you mean here. I don't think I've recommended that approach elsewhere.

Or is it better to forego that also and somehow force the tests to use the GAE Python and it's respective paths??

Our documentation shows how to use a simple test runner to do this here. You can also use NoseGAE. If you're extra curious, for testing our own samples we use py.test and a custom set of py.test hooks as well as nox with a complicated noxfile. You probably don't need all that.

ok...that makes sense....
so after creating an empty "prefix" entry in the ~/.pydistutils.cfg file, I ran this:
pip install --upgrade --requirement requirements.txt --target lib/
it put both the raw (.py files) and the matching "xxx.dist-info" package for each dependency in the lib folder.....that part is good and I should be able to delete my venv now.....and I guess I could delete those dist-info directories too to reduce my upload time??
Sadly, it also created _cffi_backend.so" in my lib directory and I know that's not allowed.....which is weird because I've not seen app engine complain when I've been pushing up there.....
Anyway, I've got to figure out which of my dependencies is pulling that one....

.and I guess I could delete those dist-info directories too to reduce my upload time??

If you really want to, but they're tiny.

Sadly, it also created _cffi_backend.so" in my lib directory and I know that's not allowed.....which is weird because I've not seen app engine complain when I've been pushing up there.....

App engine will just ignore it, so don't worry about it.

I must really be missing something because your "don't worry" comment totally worries me ;-)
How can I be certain that dependency is not in use on my localhost, and thus causing something to break when it DOES NOT get pushed to the server???

Our Import hooks prevent loading any .so module that isn't explicitly whitelisted.

Yes, I'm totally clear that it won't get loaded because it won't get pushed by the update......that's not my concern.   My concern is that the .so IS THERE in my lib/ directory when testing on localhost, and it WON'T be there on the server......so how can I trust my local testing to be definitive....in other words, some other python package depends upon it??????

I should've specified: the import hooks I mentioned are part of dev_appserver.

oh....all the worry evaporated ;-)  Thank you!

For what it's worth, if anyone else is still having this issue after reading the thread: I solved it in my case by making sure the system-wide Python path was clean of any google.* and gcloud.* related modules, and then re-installing on my virtualenv.
To clarify, my doing python -c "import sys; print sys.path" looked like this (formatted):
['/Users/dguaraglia/Projects/wrench/.env/lib/python2.7/site-packages',
 '',
 '/Users/dguaraglia/.homebrew/bin',
 '/Library/Python/2.7/site-packages',
 '/Users/dguaraglia/.homebrew/lib/python2.7/site-packages/gax_google_pubsub_v1-0.7.9-py2.7.egg',
 '/Users/dguaraglia/.homebrew/lib/python2.7/site-packages/google_gax-0.12.0-py2.7.egg',
 '/Users/dguaraglia/.homebrew/lib/python2.7/site-packages/_pdbpp_path_hack',
 '/Users/dguaraglia/.homebrew/lib/python2.7/site-packages/six-1.10.0-py2.7.egg',
  ...
]

After I manually removed the gax_google_pubsub and google_gax packages (simply rmd the files) and re-installed gcloud on my virtualenv, things went back to working.

Thanks dguaraglia.  I got over the issue by renaming all gax files in sys.path, after making sure that timestamp_pb2.py exists in one of the paths in sys.path:
/usr/local/lib/python2.7/dist-packages/google_gax-0.12.1-py2.7.egg
/usr/local/lib/python2.7/dist-packages/gax_google_pubsub_v1-0.7.10-py2.7.egg
/usr/local/lib/python2.7/dist-packages/gax_google_logging_v2-0.7.10-py2.7.egg

I have same issue
When using
import google.protobuf; print(google.protobuf.__version__) # return following issue:
ImportError: No module named protobuf
 import gcloud; print(gcloud.__version__) # return 0.18.1

@yanikkoval Some version of some package has stomped on the google namespace.  I'm pretty sure that if you had all the packages up-to-date, you'd be fine.  Can you report the output of pip freeze here?

@tseaver It's App Engine weirdness actually. I try to stay away and let @jonparrott weigh in on such issues

@yanikkoval how are you installing packages? What's in your appengine_config.py?

https://github.com/google/protobuf/issues/1153

@tseaver, @jonparrott installing it with pip put the package in the wrong directory.

@yanikkoval can you give us more details? What exact pip command are you running? What's in your appengine_config.py?

It is also happening on local machine using Ubuntu 14.04 LTS - from pip freze:
gapic-google-cloud-datastore-v1==0.14.1
gapic-google-cloud-logging-v2==0.90.1
gapic-google-cloud-pubsub-v1==0.14.1
gcloud==0.18.3
google-cloud==0.22.0
google-cloud-bigquery==0.22.1
google-cloud-bigtable==0.22.0
google-cloud-core==0.22.1
google-cloud-datastore==0.22.1
google-cloud-dns==0.22.0
google-cloud-error-reporting==0.22.0
google-cloud-happybase==0.22.0
google-cloud-language==0.22.2
google-cloud-logging==0.22.0
google-cloud-monitoring==0.22.0
google-cloud-pubsub==0.22.0
google-cloud-resource-manager==0.22.0
google-cloud-runtimeconfig==0.22.0
google-cloud-storage==0.22.0
google-cloud-translate==0.22.0
google-cloud-vision==0.22.0
grpc-google-cloud-datastore-v1==0.14.0
grpc-google-cloud-logging-v2==0.90.0
grpc-google-cloud-pubsub-v1==0.14.0

Traceback (most recent call last):
  File "./src/main.py", line 9, in <module>
    from google.cloud import logging, pubsub
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/__init__.py", line 18, in <module>
    from google.cloud.logging.client import Client
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/client.py", line 32, in <module>
    from google.cloud.client import JSONClient
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 21, in <module>
    from google.cloud._helpers import _determine_default_project
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_helpers.py", line 30, in <module>
    from google.protobuf import timestamp_pb2
ImportError: cannot import name timestamp_pb2

@ensonic Run import google; print(google.__path__) so see what the problem is. The usual suspect here is that you have a "bad" package breaking the google namespace. (The fact that you have the very old gcloud co-tenant with google-cloud isn't a good sign.)

@dhermes
python -c "import google; print(google.__path__)"
['/usr/local/lib/python2.7/dist-packages/google']

which ones should I get rid of - the 'grpc-google-' or 'gapic-google-'

Neither need be gotten rid of AFAIK. From there you'd want to poke around in /usr/local/lib/python2.7/dist-packages/google/__init__.py and see if it is a namespace package or otherwise (you also need to look around for (maybe bad?) .nspkg files in your site-packages


Feature Request: Allow different config options for different platforms

It would be awesome to have the ability to apply different config options to different platforms within the Kolide console. For example, I would like to keep events enabled on our Mac hosts while also disabling events on our Windows hosts. At this point in time, there is not a way to do that using Fleet.

As a temporary workaround, agents will respect flags that are defined in the local flagfile as long as those flags have not yet been defined on the server-side.

This will be supported in the new fleetctl work. See https://github.com/kolide/fleet/blob/master/docs/cli/README.md#osquery-configuration-options

#1792 merged fleetctl into master, resolving this issue for 2.0.0


Enable CircleCI testing

I have a .circleci/config.yml file in place, but this should be reviewed before deployment.
Also, we might need the recent changes to dask/distributed to be released before we enable CI.

Should I use TravisCI, instead?  The rest of the Dask projects seem to use Travis.  I’m happy to use Travis. It doesn’t matter to me.

Circle it is, then!

@mrocklin Can you add the dask-mpi project to CircleCI?  Only the project admin can do it.

I think that this is done.  https://circleci.com/gh/dask/workflows/dask-mpi
On Wed, Dec 26, 2018 at 7:50 AM Kevin Paul<EMAIL_ADDRESS>wrote:

@mrocklin https://github.com/mrocklin Can you add the dask-mpi project
to CircleCI? Only the project admin can do it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-mpi/issues/7#issuecomment-449984655, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszPsI9VR_vadHyjyTRLZ16TX8ht4Aks5u85rhgaJpZM4Zc3CC
.

(though I'm not super familiar here, and so could be wrong)
On Wed, Dec 26, 2018 at 9:03 AM Matthew Rocklin<EMAIL_ADDRESS>wrote:

I think that this is done.
https://circleci.com/gh/dask/workflows/dask-mpi
On Wed, Dec 26, 2018 at 7:50 AM Kevin Paul<EMAIL_ADDRESS>wrote:

@mrocklin https://github.com/mrocklin Can you add the dask-mpi project
to CircleCI? Only the project admin can do it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-mpi/issues/7#issuecomment-449984655, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszPsI9VR_vadHyjyTRLZ16TX8ht4Aks5u85rhgaJpZM4Zc3CC
.


wechat icon

Proposed Changes

Add wechat icon

Screenshot (If Applicable)
Checklist

[ ] Tested locally
[ ] Ran yarn ci to test my code
[ ] Did not add any unnecessary changes
[ ] All my changes appear after other changes in each file
[ ] Included a screenshot (if adding a new button)
[ ] 🚀

Hey thanks!  I think we will need more code than this to merge it.  Look for an example in a previous PR!

Just rechecked. Wechat released an update 6.3.25 on 2016-09-05, which blocks almost all URL schemes request from non-weixin's application. Only requests from weixin's applications and some "white-listed" applications are allowed.
Hence normal browser application doesnt execute the url scheme now. Will have to relook into this if there's any changes


Log4j2 throws UnsupportedOperationException

Background info:
Vertx 4.5.1
slf4j 2.0.9
log4j 2.21.0
reactiverse-contextual-logging 1.1.2
Problem:
When logging to the console, the following stacktrace is produced:
2024-01-11T12:59:34,943 INFO  HttpOutLogHandler - [vert.x-eventloop-thread-1] - [exchangeId:66a74e3b-40eb-4ac8-80f5-4ae27ec3a9f9, port:20001, uri:/v1/management/version, client:0:0:0:0:0:0:0:1]
  HTTP response code=200
WARN StatusConsoleListener org.apache.logging.slf4j.Log4jEventBuilder caught java.lang.UnsupportedOperationException logging ReusableParameterizedMessage: HTTP response code=200
 java.lang.UnsupportedOperationException
        at java.base/java.util.Collections$UnmodifiableMap.clear(Collections.java:1668)
        at org.apache.logging.log4j.core.impl.JdkMapAdapterStringMap.clear(JdkMapAdapterStringMap.java:123)
        at org.apache.logging.log4j.core.impl.MutableLogEvent.clear(MutableLogEvent.java:144)
        at org.apache.logging.log4j.core.impl.ReusableLogEventFactory.release(ReusableLogEventFactory.java:130)
        at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:559)
        at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:81)
        at org.apache.logging.log4j.core.Logger.log(Logger.java:163)
        at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2066)
        at org.apache.logging.log4j.internal.DefaultLogBuilder.logMessage(DefaultLogBuilder.java:250)
        at org.apache.logging.log4j.internal.DefaultLogBuilder.log(DefaultLogBuilder.java:144)
       ...

After analysis, I've found that the problem seems to be related to this library. More specifically, the class io.reactiverse.contextual.logging.VertxContextDataProvider creates an immutable map that the logging framework later attempts to clear, causing the problem. This stacktrace is repeated for every line that is subsequently logged to the console. I tried creating my own instance of the VertxContextDataProvider with a regular HashMap instead of an immutable collection and the problem seems to be resolved. My problem is that I don't know whether I'm introducing side-effects by changing the data structure accessibility.
Here is the log4j xml file configuration used to produce the problem:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration
        status="warn"
        strict="true"
        name="some-name">
    <Properties>
        <Property name="ServiceSpace" value="some-space"/>
        <Property name="ServiceName" value="some-name"/>
        <Property name="ExecutionEnvironment" value="${env:EXECUTION_ENVIRONMENT}"/>
        <Property name="Layout" value="%style{%d{ISO8601}}{black} %highlight{%-5level} %style{%C{1}}{bright,yellow} - [%t]%notEmpty{ - [exchangeId:%X{exchangeId}, port:%X{port}, uri:%X{uri}, client:%X{clientAddress}]}%n  %style{%msg}{bright}%n%throwable"/>
        <Property name="FileLayout" value="%d{ISO8601} %-5level %C{1} - [%t]%notEmpty{ - [exchangeId:%X{exchangeId}, port:%X{port}, uri:%X{uri}, client:%X{clientAddress}]}%n  %msg%n%throwable"/>
    </Properties>
    <Filter type="ThresholdFilter" level="trace"/>

    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="${Layout}"/>
        </Console>

        <RollingFile name="RollingFile"
                     fileName="./logs/${ServiceName}.log"
                     filePattern="./logs/$${date:yyyy-MM}/${ServiceName}-%d{yyyy-MM-dd}-%i.log.gz">
            <PatternLayout pattern="${FileLayout}"/>
            <Policies>
                <!-- rollover on startup, daily and when the file reaches
                    10 MegaBytes -->
                <OnStartupTriggeringPolicy />
                <SizeBasedTriggeringPolicy
                        size="10 MB" />
                <TimeBasedTriggeringPolicy />
            </Policies>
        </RollingFile>

        <RollingFile name="RollingFileJson"
                     fileName="./logs/${ServiceName}-json.log"
                     filePattern="./logs/$${date:yyyy-MM}/${ServiceName}-%d{yyyy-MM-dd}-%i-json.log.gz">
            <JsonTemplateLayout
                    eventTemplateUri="classpath:mcmopiadapter-json-layout.json"
                    stackTraceEnabled="true"
                    locationInfoEnabled="false"
            >
                <EventTemplateAdditionalField key="service-space" value="${ServiceSpace}"/>
                <EventTemplateAdditionalField key="service-name" value="${ServiceName}"/>
                <EventTemplateAdditionalField key="service-environment" value="${ExecutionEnvironment}"/>
            </JsonTemplateLayout>

            <Policies>
                <!-- rollover daily and when the file reaches 10 MegaBytes -->
                <SizeBasedTriggeringPolicy size="10 MB"/>
                <TimeBasedTriggeringPolicy interval="1" modulate="true"/>
            </Policies>
        </RollingFile>
    </Appenders>

    <Loggers>
        <Logger name="io.micrometer" level="INFO"/>
        <Logger name="io.netty" level="INFO"/>
        <!-- LOG everything at INFO level unless overriden with -DlogLevel=... -->
        <Root level="${env:LOG_LEVEL:-INFO}">
            <AppenderRef ref="Console" />
            <AppenderRef ref="RollingFile" />
            <AppenderRef ref="RollingFileJson"/>
        </Root>

    </Loggers>

</Configuration>

@BenjaminTanguay I've sent PR https://github.com/reactiverse/reactiverse-contextual-logging/pull/98 which should solve the issue. Can you give it a try?

@BenjaminTanguay I'm asking because I wasn't able to reproduce locally

I'll check...

@tsegismont The fix seems to work. Thank you for the help!

same problem, thank u guys

I am concerned that I wasn't able to reproduce. I can see how the previous implementation did not conform to log4j2 expectations, but I would prefer to have a test to add to this PR.
Is any of you able to provide a simple reproducer ?

I tried all day creating a small reproducer based on our production code and I still can't do it. I tried a bunch of permutations but I can't isolate the factor responsible for triggering the bug. I can't justify working on this for much longer so I'll just stop. I may or may not come back to this on my own time.

Fixed in version 1.2.1 (on its way to Maven Central)


Need consideration about resources lifetime

RCL API has a order to resource initialization and finalization. For example, rcl_shutdown needs to be called before rcl_context_fini. In order to handle it, an architecture to manager resource initialization, running and finalization is required.

Fixed in #22 .
Made Context struct and call terminate of Nodes at destructor.


Error with apcu

Hello,
I am running Ubuntu 14.04 and I am using this buildpack but I am getting an error.
-----> Bundling extensions
       apcu
remote:
remote: gzip: stdin: not in gzip format
remote: tar: Child returned status 1
remote: tar: Error is not recoverable: exiting now

Any idea of what is wrong? It seems to not like the apcu. If I look in the cache directory the file(s) it is only 70 kb's and it says
Not FoundThe resource could not be found.
Both phpredis and apcu seem to fail. mongo is fine.
I can manually fix it by going into the cache directory and wget'ing the file manually.

Thank you for reporting the error, I'll fix that asap !

-----> Bundling NGINX 1.6.2
-----> Bundling PHP 5.6.4
-----> Bundling extensions
       apcu
       phpredis
       mongo
-----> Setting up default configuration
-----> Vendoring binaries into slug

No problem with me.
Which PHP version are you using?

lol that was super fast. I am using PHP 5.3.28 and a PHP (classic) app.

-----> Use /tmp/b for build
-----> Bundling NGINX 1.6.2
       Checksums match. Fetching from cache.
-----> Bundling PHP 5.3.28
-----> Bundling extensions
       apcu

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
 !     An error occured during buildpack compilation

Ok I got it

Ok it should work now, reopen if not ;-)