How to enable amazon S3 storage for all subdomains

Hi,
Please let me know how to enable to amazon S3 storage for all subdomains, currently I enabled S3 storage on main site, but still subdomains are using local storage, so I wanted all subdomains also use same S3 storage as main site so that there is no need to configure subdomains manually , please let me know how I can do that.

you can use the hook "on_uploader" or overwrite this method by your initializer.
https://github.com/owen2345/camaleon-cms/blob/master/app/helpers/camaleon_cms/uploader_helper.rb#L132

Thank you, I will try to explore more, but  it would be great if you can provide some sudo/real code to start with, Thanks

Sample using a hook (create a plugin):
add the hook in your my_plugin/config/camaleon_plugin.json
"hooks": {
    ....
    "on_uploader": [
      "my_plugin_my_method_name"
    ]
    ...

add this method in your plugin helper (my_plugin/app/helpers/plugins/my_plugin/main_helper.rb):
def my_plugin_my_method_name(args)
  main_site = CamaleonCms::Site.main_site
  args[:connection] = Fog::Storage.new({ :aws_access_key_id =>main_site.get_option("filesystem_s3_access_key"), :provider   => 'AWS', aws_secret_access_key: main_site.get_option("filesystem_s3_secret_key"), :region  => main_site.get_option("filesystem_region") })
  args[:bucket_name] = main_site.get_option("filesystem_s3_bucket_name")
end

Thanks a lot, I will try and update you. Thank you.

Hi,
I added system hook and created helper method as you suggested, now its working fine for all my subdomains.
However its taking time to load image upload window when I click on upload image, I believe its due to the fact that its trying to load all already existing images from S3, but it may create problem if there are several images.
Any way once again thanks for the solution.

great!
about the slow problem I will improve in next versions.
PD: please don't forget to share camaleon by social networks.

Sure, I will do that, its great CMS..

Closing this issue because now I am able to use S3 for all subdomains as mentioned above. Thanks

Hi @owen2345 ,
With latest stable master branch, I am unable to use S3 for all subdomains though I added below helper method in "ApplicationHelper" file.
 def aws_s3_for_all_subdomains(args)
    main_site = CamaleonCms::Site.main_site
    args[:connection] = Fog::Storage.new({ :aws_access_key_id =>main_site.get_option("filesystem_s3_access_key"), :provider   => 'AWS', aws_secret_access_key: main_site.get_option("filesystem_s3_secret_key"), :region  => main_site.get_option("filesystem_region") })
    args[:bucket_name] = main_site.get_option("filesystem_s3_bucket_name")
  end

I also modified system.json file as below

"on_uploader": [
      "aws_s3_for_all_subdomains"
    ]
  }

As far I remember this was working fine earlier, but not sure from when it's broken.

Hi @mazharoddin ,
I am sorry, but I removed the fog dependency for many problems with file permissions and cache, the easy way to do the same is by overwriting the following method:
https://github.com/owen2345/camaleon-cms/blob/master/app/uploaders/camaleon_cms_aws_uploader.rb#L74
Regards,

You could subclass it maybe instead of overwriting it?
class YourUploader < CamaleonCmsAwsUploader

Like this:
CamaleonCmsAwsUploader.class_eval do
  def bucket
  # here overwrite the current function
  end
end

np @owen2345, I am able to upload images to s3 and cloudfront using AWS uploaded, but still sub domains are not able to use main site S3 bucket details.
I modified earlier application helper as below.
module ApplicationHelper
  def aws_s3_for_all_subdomains(args)
    main_site = CamaleonCms::Site.main_site
    args[:connection] = Aws.config.update({ region: main_site.get_option("filesystem_region"),credentials: Aws::Credentials.new(main_site.get_option("filesystem_s3_access_key"),main_site.get_option("filesystem_s3_secret_key"))})
     s3 = Aws::S3::Resource.new
    args[:bucket_name] = s3.bucket(main_site.get_option("filesystem_s3_bucket_name"))
  end
end

Please let me know if any changes are required. Thanks

Hi @owen2345 ,
Can you please help me writing this helper method, due to this issue sub sites are using local space, but I wanted them to use main site s3 and cloudfront IDs.

Hi @mazharoddin
you need to overwrite this method:
CamaleonCmsAwsUploader.class_eval do
  def bucket
    @bucket ||= lambda{
      main_site = Cama::Site.main_site
      config = Aws.config.update({ region: main_site.get_option("filesystem_region", 'us-west-2'), credentials: Aws::Credentials.new(main_site.get_option("filesystem_s3_access_key"), main_site.get_option("filesystem_s3_secret_key")) })
      s3 = Aws::S3::Resource.new
      bucket = s3.bucket(main_site.get_option("filesystem_s3_bucket_name"))
    }.call
  end
end

Thanks @owen2345

Hi @owen2345,
While testing this change I found that settings are not getting saved, I am unable to upload new logo or change media settings etc with latest code.
I am using  camaleon_cms 2.2.1 from git://github.com/owen2345/camaleon-cms.git.
Can you please check the issue.

@owen2345, I am unable to save S3/cloudfront settings as mentioned above, when i click save button I get confirmation mentioning site is updated, but settings are not getting saved.

Fixed, please use 2.3.x because that version includes a typing error.
Regards,

Ok, Thanks @owen2345 , I will update once I test it.

@owen2345 , now main site settings are getting successfully, Thank you!!.
But still I am unable to use main site S3 and cloudfront urls with other subdomains, I tried to override "CamaleonCmsAwsUploader" class using below code, I tried several time but its not working.
I created config/initializers/camaleon_cms.rb  file and added below code as you suggested.
CamaleonCmsUploader::CamaleonCmsAwsUploader.class_eval do
  # initialize a bucket with AWS configurations
  # return: (AWS Bucket object)
  def bucket
    @bucket ||= lambda{
      main_site = Cama::Site.main_site

      config = Aws.config.update({ region: main_site.get_option("filesystem_region", 'ap-southeast-1'), credentials: Aws::Credentials.new(main_site.get_option("filesystem_s3_access_key"), main_site.get_option("filesystem_s3_secret_key")) })
      s3 = Aws::S3::Resource.new
      bucket = s3.bucket(main_site.get_option("filesystem_s3_bucket_name"))
    }.call
  end
end

Sorry for re-opening same issue again.

Hi @mazharoddin,
could you please tell me what is the problem?
is there any error in the console? or is not calling to your custom method?
Regards,

hi @owen2345 ,
There are no errors in console, i believe its not calling my custom method..one more thing, earlier I used to load "aws_s3_for_all_subdomains" using on_uploader hook, but currently I am not using this hook since I am overriding  "CamaleonCmsAwsUploader" class in initialize.
So looks like subdomain not loading bucket config without selecting "s3" storage option in subdomain settings here.

hi @owen2345 ,
Can you please let me know how i can use old code in 2.3.3, atlease I will try to replace earlier working code, due to this issue I am not loading any images on subdomains, because I wanted all subdomains use s3 storage instead of local storage..Thanks

Hi @mazharoddin
I re added the hook to customize file uploader from hooks:
args={ server: 'local/aws', thumb: {w: 100, h: 100}, aws_settings: {region: 'us-west-2', access_key: 'xxx', secret_key: 'yyy', bucket: 'test'}}; hooks_run("on_uploader", args)

Please let me know if this works for you.
Regards,

Let me try it, thank you @owen2345

Hi @owen2345, can you please let me know how to test this change..

create a hook

add in config.json

" hooks": {
"on_uploader": ["my_helper"]
}

in my helper.rb

def my_helper(args)
  args[:server] = 'aws'
  args[:aws_settings] = {region: 'us-west-2', access_key: 'xxx', secret_key: 'yyy', bucket: 'test'}
end

restart server and all file conections will use this setting

Thanks @owen2345 ,
After some testing, I could able to use S3 for subdomains with below code.
 def aws_s3_for_all_subdomains(args)
    main_site = CamaleonCms::Site.main_site
    args[:aws_settings] = Aws.config.update({ region: main_site.get_option("filesystem_region"),credentials: Aws::Credentials.new(main_site.get_option("filesystem_s3_access_key"),main_site.get_option("filesystem_s3_secret_key"))})
    s3 = Aws::S3::Resource.new
    args[:bucket] = s3.bucket(main_site.get_option("filesystem_s3_bucket_name"))

  end

But I will change as you suggested..Thanks a lot

great!
be careful with "aws_settings" , it receives a hash

ohh Ok, Sure, but some reason I am unable to upload images to S3 for subdomain with your suggested helper...I am testing further ...I am trying to test cloudfront aswell... I will update you once I test S3 and cloudfront with subdomains...thank you ...

Hi @owen2345,
I tested various combinations, but only below code is working.
 def aws_s3_for_all_subdomains(args)
    args[:server] = 's3'
    main_site = CamaleonCms::Site.main_site
    args[:aws_settings] = Aws.config.update({ region: main_site.get_option("filesystem_region"),credentials: Aws::Credentials.new(main_site.get_option("filesystem_s3_access_key"),main_site.get_option("filesystem_s3_secret_key"))})
    s3 = Aws::S3::Resource.new
    args[:bucket] = s3.bucket(main_site.get_option("filesystem_s3_bucket_name"))
  end

Also note that for reason only args[:server] = 's3' is working instead of args[:server] = 'aws'
Onemore thing I am unable to use main site cloudfront url for subdomains, without that subdomains will use lot of bandwidth of S3, moreover images will load very slowly, can you please add cloudfront support aswell...thank you..

it is strange your definition because this is function which is using your configuration:
https://github.com/owen2345/camaleon-cms/blob/master/app/uploaders/camaleon_cms_aws_uploader.rb#L76
Also I added the fix for cloudfront customization

Not sure how, but its working :) ..ok let me test the cloudfront part ..thanks..one more thing, I am seeing performance issue while main site user is trying to delete a image and if subdomain user trying delete/add new image, site completely becoming unusable for some time.
I will further test and update you..Thanks for the cloudfront fix...

Hi @owen2345,
Cloudfront hook is not working, if I enabled cloud front in main site and if I didn't change earlier helper hook then I am getting below error if I try to access subdomain media folder.
undefined method `objects' for nil:NilClass Did you mean? object_id
    objects = {}
    objects['/'] = {files: {}, folders: {}}
    bucket.objects.each do |file|
      cache_item(file_parse(file), objects)
    end
    @current_site.set_meta(cache_key, objects)
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails4.2.7/bundler/gems/camaleon-cms-5d1e0b205798/app/uploaders/camaleon_cms_aws_uploader.rb:6:in `browser_files'
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails4.2.7/bundler/gems/camaleon-cms-5d1e0b205798/app/uploaders/camaleon_cms_uploader.rb:23:in `objects'
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails4.2.7/bundler/gems/camaleon-cms-5d1e0b205798/app/controllers/camaleon_cms/admin/media_controller.rb:84:in `init_media_vars'

I am getting below error if added cloud_front option to helper function.
invalid configuration option `:cloud_front'
    main_site = CamaleonCms::Site.main_site
    args[:aws_settings] = Aws.config.update({ region: main_site.get_option("filesystem_region"),cloud_front: main_site.get_option("filesystem_s3_cloudfront"), credentials: Aws::Credentials.new(main_site.get_option("filesystem_s3_access_key"),main_site.get_option("filesystem_s3_secret_key"))})
    s3 = Aws::S3::Resource.new
    args[:bucket] = s3.bucket(main_site.get_option("filesystem_s3_bucket_name"))
  end

app/helpers/application_helper.rb:6:in `new'
app/helpers/application_helper.rb:6:in `aws_s3_for_all_subdomains'

I also tried changing helper as below.
def aws_s3_for_all_subdomains(args)
    args[:server] = 's3'
    main_site = CamaleonCms::Site.main_site
    args[:aws_settings] = Aws.config.update({ region: main_site.get_option("filesystem_region"), credentials: Aws::Credentials.new(main_site.get_option("filesystem_s3_access_key"),main_site.get_option("filesystem_s3_secret_key"))})
    s3 = Aws::S3::Resource.new
    args[:bucket] = s3.bucket(main_site.get_option("filesystem_s3_bucket_name"))
    args[:cloud_front] = main_site.get_option("filesystem_s3_cloudfront")
  end

Hi @owen2345,
if possible can you please have a look into this issue once. Thanks.

Hi @mazharoddin,
sorry for my delay, I found the problem (there was || instead a ,).
Fixed and tested with:
def my_helper(args)
    args[:server] = 'aws'
    args[:aws_settings] = {region: 'us-west-2',
                           access_key: '0909090909',
                           secret_key: '01010101010101010100101100',
                           bucket: 'my_bucket',
                           cloud_front: 'https://d199wmwx3434.cloudfront.net'}
  end

Note: this hook should be register in your system.json to be executed for all sites.
Note2: Don't forget to "clear cache" in media browser
Test it using git version.
Regards,

np @owen2345 , its working now, thank you so much...
One problem observed is if I change filesystem setting from local to aws and go back to local then all files are getting deleted, though I wont be using local storage, but it may create problem for users who uses local storage, so thought to report it.
I can open seperate issue if required. Thanks.

Hi @owen2345 ,
Sorry to report it, but I am seeing one more issue related to S3 after deploying fix to staging server.
F, [2016-09-03T12:23:41.101755 #9510] FATAL -- : NoMethodError (undefined method `objects' for nil:NilClass
Did you mean?  object_id):
F, [2016-09-03T12:23:41.102293 #9510] FATAL -- :
F, [2016-09-03T12:23:41.103130 #9510] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_aws_uploader.rb:15:in `browser_files'
F, [2016-09-03T12:23:41.103636 #9510] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:23:in `objects'
F, [2016-09-03T12:23:41.104067 #9510] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/controllers/camaleon_cms/admin/media_controller.rb:87:in `init_media_vars'
F, [2016-09-03T12:23:41.104454 #9510] FATAL -- : activesupport (5.0.0) lib/active_support/callbacks.rb:382:in `block in make_lambda'

Due to this issue I am getting 500 error if I try to access media folder on subdomain.

Looks like there is some issue with cache, I am getting below error on my local setup aswell.
        "thumb" => '',
        'type' => is_dir ? '' : (s3_file.content_type rescue (MIME::Types.type_for(key).first.content_type rescue "")),
        'created_at' => is_dir ? '' : s3_file.last_modified,
        'dimension' => ''
    }.with_indifferent_access
    res["thumb"] = version_path(res['url']) if res['format'] == 'image' && File.extname(res['name']).downcase != '.gif'

/Users/mazhar/.rvm/gems/ruby-2.3.0@rails5.0.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_aws_uploader.rb:37:in `file_parse'
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails5.0.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `block in cache_item'
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails5.0.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `each'
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails5.0.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `each_with_index'
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails5.0.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `cache_item'
/Users/mazhar/.rvm/gems/ruby-2.3.0@rails5.0.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_aws_uploader.rb:16:in `block in browser_files'

I think that the problem for "if I try to access media folder on subdomain" is because your hook is not calling for sub domains, put your hook in system.json and the function in applicatio helper.

About the cache error, after you changed the file storage settings, you need to clear cache to re scan the files.

Hi,
No I put the the hook in system.json and in application helper aswell
yes, I tried clearing the cache, dropping the database and recreating etc, but I am seeing same issue.

@mazharoddin, Check my image, I don't have any error:

Tried many times without any error.
Regards,

ohh ok, sure I will try again, but it was not working for me even on newly created site aswell.

hi @owen2345 , for some reason its not fully working for me, it worked for some time if I restart the server by changing filesystem back to local and back to S3, but  if I create new subdomain then I am seeing same issue again.
This time i got slightly different error.
F, [2016-09-04T15:11:50.167707 #6669] FATAL -- : NoMethodError (undefined method `last_modified' for "/production/database/production_backup/2<IP_ADDRESS>.47.44":String):
F, [2016-09-04T15:11:50.168159 #6669] FATAL -- :
F, [2016-09-04T15:11:50.168415 #6669] FATAL -- : /home/deployer/apps/shop
/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_aws_uploader.rb:37:in `file_parse'
F, [2016-09-04T15:11:50.168773 #6669] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `block in cache_item'
F, [2016-09-04T15:11:50.169290 #6669] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `each'
F, [2016-09-04T15:11:50.169534 #6669] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `each_with_index'
F, [2016-09-04T15:11:50.170201 #6669] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_uploader.rb:59:in `cache_item'
F, [2016-09-04T15:11:50.171297 #6669] FATAL -- : /home/deployer/apps/shop/shared/bundle/ruby/2.3.0/bundler/gems/camaleon-cms-34f309c45780/app/uploaders/camaleon_cms_aws_uploader.rb:16:in `block in browser_files'
F, [2016-09-04T15:11:50.172310 #6669] FATAL -- : aws-sdk-resources (2.5.8) lib/aws-sdk-resources/batch.rb:75:in `each'

I think this error occurs if we had some previous S3 images and later that subdomain settings were changed to use local. Now if I  change back to S3 then I am hitting above issue.

@mazharoddin could you please send me a video or steps detailed to reproduce the problem?
PD: I tried many times to reproduce the problem but without success.
Regards,

Sure @owen2345

Hi @owen2345,
Please find below for the video with error while accessing media, which I could able to create on clean repository.
https://sendvid.com/yn94ko1b#

I checked in the changes to below repository. Thanks.
https://github.com/mazharoddin/camaleon_cms_debug

Hi @owen2345,
Please let me know if you could able to find out the issue.

Hi @mazharoddin,
Sorry for the delay but I was too busy these days, the problem was because you have a file within multiple subfolders in AWS, like:
/foder1/folder2/folder3/my_file.jpg
I fixed the problem. please let me if it works well for you too.
Regards,

Sure @owen2345 np, I will test and let you know ..thanks a lot.

Hi @owen2345 ,
Now I am not seeing that issue, Thanks for the fix, but now all subdomains  showing same images across the site, so all subdomains are showing root domian + other subdomains images ...so it creating privacy problem :(

Hi @mazharoddin,
For now the AWS Uploader is waiting one bucket for each sub site.
Regards,

ohh ok, but in hook we are specifying bucket name, since we wanted  these settings to applicable for all subdomain, so they are going to applicable for all sites, so we don't have any option to specify bucket name for each sub site.
Pls do let me know we can specify bucket name for each subdomain, if its not possible, then the idea of sharing AWS settings across subdomain may not be useful, because each subdomain is different and they are owned by different users, so we may need to find a way use same bucket and store files by creating separate directory for each subdomain, or creating bucket for each subdomain during subdomain creation. what do you feel?.
args[:aws_settings] = {  region:     ENV["AWS_REGION"],
                             access_key: ENV["AWS_ACCESS_KEY_ID"],·
                             secret_key: ENV["AWS_SECRET_ACCESS_KEY"],·
                             bucket:     ENV["S3_BUCKET_NAME"],·

Hi @owen2345 ,
During my testing I observed that few subdomains have only images which I uploaded from subdomain have, but other subdomains are showing all images from the bucket, both are sharing same s3 config, so not sure why only few showing images to that specific subdomain uploaded images.

Please try cleaning cache in media section to fetch again the files from aws.

yeah I tried, but still it was showing all images for that subdomain,  I will do one thing, I will try create new bucket and try to assign it and test..thanks.

Hi @owen2345 ,
I tried with other bucket, but I am seeing same issue with new bucket aswell, bucket images are shown across the site, and any one can add or delete images from the bucket..since we are overriding S3 settings using hook, I am unable to add buckets specif to subdoman and there is a limitation with bucket names at S3 ..

Hi @owen2345,
If possible can you please provide some sudo code on using single bucket across the subdomains, but restricting access to its own subdomain photos/ directories, I will try to implement it if possible.

Hi @mazharoddin
I added the support to manage your situation, sample:
def uploader_hook_for_aws(args)
  args[:aws_settings]['inner_folder'] = 'my_folder'
end

Please test using git version.
Regards,

hi @own2345,
wow really great, thanks, I will try it, but looks like the folder will be shared across all subdomains? but each subdomain images shoud't be visible to other subdomain since camaleon_cms is multi site...so will check if that works are not ..thank you so much for looking into it.

@owen2345 , I tried to add new hook as you suggested, but its not working. Can you please check if you merged this code as I am unable to find any commit related to this.
Is its possible to create a folder whenever we create a subdomain and restrict access to that folder for that subdomain and later save all files related to that subdomain only in that folder.
That way I think we can have different subdomain files in same bucket. What do you feel.

Hi @mazharoddin ,
I am so sorry, my changed was not pushed to the master branch because there were conflicts.
Please try again:
https://github.com/owen2345/camaleon-cms/blob/master/app/uploaders/camaleon_cms_aws_uploader.rb#L23
Regards,

Sure, Thanks @owen2345

Hi @owen2345,
Now images are uploaded to specified folder, but since we are overriding aws configuration using application helper, its applicable to all entire site, so now still all subdomains are able to access the images from 'my_folder', so still the actual issue persists, I think you can fix this issue by creating folders for each subdomain and restricting access to that folder based on subdomain...

@mazharoddin did you put a custom name for each site? like:
{"inner_folder" => "my_folder_#{current_site.id}"}

By this way you are assigning an unique folder for each site.
Regards,

Yeah @owen2345, now its working as expected, thanks a lot for the fix. I will close this issue after some more testing.

THANK YOU FOR THIS CHANGES


Higher level API for building console applications

Is your feature request related to a problem? Please describe.
The lack of a library that allows me to easily build a console application by just adding commands.
Describe the solution you'd like
Something very similar to this
Describe alternatives you've considered
mosaic
Additional context
For building CLI tooling this library is still very low-level, it focuses on the input/output formatting but that is a secondary concern when you just want to quickly build a CLI application.

I did find https://ajalt.github.io/clikt/quickstart/, that does what I am looking for.


03.07-Merge-and-Join references data files that do not exist.

Number of csv files missing from repo.
IOErrorTraceback (most recent call last)
<ipython-input-20-0928a7e92a5e> in <module>()
----> 1 pop = pd.read_csv('state-population.csv')
      2 areas = pd.read_csv('state-areas.csv')
      3 abbrevs = pd.read_csv('state-abbrevs.csv')
      4
      5 display('pop.head()', 'areas.head()', 'abbrevs.head()')

Okay, so am zipping through and not reading that need to download these manually.
# Following are shell commands to download the data
# !curl -O https://raw.githubusercontent.com/jakevdp/data-USstates/master/state-population.csv
# !curl -O https://raw.githubusercontent.com/jakevdp/data-USstates/master/state-areas.csv
# !curl -O https://raw.githubusercontent.com/jakevdp/data-USstates/master/state-abbrevs.csv

Would be better if notebook could pull them down itself.


enable the execution of encrypted javascript code

Hi,
in one of my projects I wanted to include small javscript code in an encrypted page which shouldn't be readable without knowing the password. So reload_js was not an option, besides it requires jquery which I do not use in my theme.
I made some small changes to find a script id and eval it on successful decryption:
plugin.py
@@ -74,6 +74,7 @@ class encryptContentPlugin(BasePlugin):
         ('encrypted_something', config_options.Type(dict, default={})),
         ('search_index', config_options.Choice(('clear', 'dynamically', 'encrypted'), default='encrypted')),
         ('reload_scripts', config_options.Type(list, default=[])),
+        ('eval_script_id', config_options.Type(string_types, default='')),
         ('experimental', config_options.Type(bool, default=False)),
         # legacy features, doesn't exist anymore
         ('disable_cookie_protection', config_options.Type(bool, default=False)),
@@ -137,6 +138,7 @@ class encryptContentPlugin(BasePlugin):
             'default_expire_dalay': int(self.config['default_expire_dalay']),
             'encrypted_something': self.config['encrypted_something'],
             'reload_scripts': self.config['reload_scripts'],
+            'eval_script_id': self.config['eval_script_id'],
             'experimental': self.config['experimental']
         })
         return decrypt_js

decrypt-contents.tpl.js
@@ -182,6 +182,12 @@ function decrypt_action(password_input, encrypted_content, decrypted_content) {
             reload_js(reload_scripts[i]);
         }
         {%- endif %}
+        {% if eval_script_id != '' -%}
+        let eval_script = document.getElementById("{{ eval_script_id }}");
+        if (eval_script) {
+            eval(eval_script.innerHTML);
+        }
+        {%- endif %}
         return true
     } else {
         // create HTML element for the inform message

here is an example configuration:
eval_script_id_example.zip
It's a bit tricky, because if you simply declare a function it won't be usable by f.ex. onClick events. the function needs to be declared in a window.newfunction = function() { ... } style, but see the example.
I'll just leave it here as suggestion. It's probably not the most elegant way (because it is a bit tricky and required adjusting the javascript code) but i works for me.
Another suggestion would be to rewrite the reload_js function not to use jquery.

Regarding your remarks about this function, I haven't found any acceptable ways to do what you want.
Using eval() may be the correct method, but... I don't want to use it.
I therefore decide not to implement your proposal for the moment (in the hope of finding something better).
However, you can implement your reload function yourself, by using the new override default templates in Version 2.3.0.

you are right, the use if eval is not a good practice.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval?retiredLocale=de#never_use_eval!


Fourth Answer

Comparing the develop branch with the fourth file's answer.

Merged develop branch with fourth answer branch.


Add persistent per request degradation on failure

When bad things with redis happens, it raises a lot of Timeout or Connection error. If DEGRADE_ON_FAILURE is enabled, django goes to database after every error and can get its data anyway. But for every db request it does three (and even four if redisreplica is used) queries: two for asking replica and master for a cached data, one for getting data from db, and another one for saving cache.
In this fix client will be marked as degraded if Timeout or Connection error happens. After that no queries will be executed by this client during current request. So in case which described above we will have only three queries per first fetch (asking replica and master for cached data and getting data from db), and we won't try to ask cacheops for all next queries untill request won't be finished.
It's disabled by default and can be enabled by setting CACHEOPS_DEGRADE_PERSISTENT_PER_REQUEST flag to True.

@ttyS15, @tumb1er I think it can be interesting for you


Update scalatest to 3.2.15

Updates org.scalatest:scalatest from 3.2.11 to 3.2.15.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Files still referring to the old version number
The following files still refer to the old version number (3.2.11).
You might want to review and update them manually.
build.sbt

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scalatest", artifactId = "scalatest" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "org.scalatest", artifactId = "scalatest" }
}]

labels: test-library-update, early-semver-patch, semver-spec-patch, old-version-remains, commit-count:1

Codecov Report
Base: 41.66% // Head: 38.88% // Decreases project coverage by -2.77% :warning:

Coverage data is based on head (34a2b7c) compared to base (90e05bb).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #273      +/-   ##
==========================================
- Coverage   41.66%   38.88%   -2.78%
==========================================
  Files           1        1
  Lines          36       36
  Branches        1        1
==========================================
- Hits           15       14       -1
- Misses         21       22       +1

Impacted Files
Coverage Δ

.../main/scala/io/circe/bson/BsonCodecInstances.scala
38.88% <0.00%> (-2.78%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Superseded by #282.


Very slow performance when using the proxy

Bug Description
I am running load test before trying it in production, and I noticed a very huge performance overhead when using the proxy.
How to reproduce

Create cloud sql proxy with the following config:

Create a test VM - e2-standard-16 (16 vCPUs, 64 GB memory)
Run cloud sql proxy as a process (using the latest version binary).

Results:
pgbench -c 500 -j 2 -t 10000 postgres -h <> // directly using the public ip
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = "C.UTF-8"
  are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("C.UTF-8").
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 500
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 5000000/5000000
latency average = 227.917 ms
tps = 2193.783436 (including connections establishing)
tps = 2193.795428 (excluding connections establishing)

$ pgbench -c 500 -j 2 -t 10000 postgres. // with the proxy
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = "C.UTF-8"
  are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("C.UTF-8").
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 500
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 5000000/5000000
latency average = 343.441 ms
tps = 1455.854525 (including connections establishing)
tps = 1455.863860 (excluding connections establishing)

Is this expected? Maybe something I mis-configured?
Environment

OS type and version: Ubuntu
Cloud SQL Proxy version (./cloud_sql_proxy -version): Cloud SQL Proxy: 1.18.0

Hi @omerlh, thanks for opening the issue! We're working on reproducing it and will follow up soon.

To follow up, we were able to replicate the numbers you were seeing when using the same settings for pgbench as you had used above.
Command: pgbench -c 500 -j 2 -t 10000 postgres -h [instance public IP] -U postgrespgbench -c 500 -j 2 -t 10000 postgres -h [instance public IP] -U postgres
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 500
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 5000000/5000000
latency average = 212.110 ms
tps = 2357.271871 (including connections establishing)
tps = 2357.282742 (excluding connections establishing)

Command: pgbench -c 500 -j 2 -t 10000 postgres -h <IP_ADDRESS> -U postgres
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 500
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 5000000/5000000
latency average = 339.099 ms
tps = 1474.496186 (including connections establishing)
tps = 1474.504326 (excluding connections establishing)

After that, we tested a few variables that could have caused the overhead
.

First, we tried running pgbench using SSL, and comparing it to both the proxy connection and the direct public IP connection. The SSL connection took slightly longer (around 218ms, compared to the 212ms for public IP without SSL), but that difference was negligible compared to the large overhead we were seeing for the proxy.

Command:
pgbench -c 500 -j 2 -t 10000 "sslmode=verify-ca \
sslrootcert=server-ca.pem \
sslcert=client-cert.pem sslkey=client-key.pem \
hostaddr=[Public IP Address] \
user=postgres dbname=postgres"

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 500
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 5000000/5000000
latency average = 218.770 ms
tps = 2285.503365 (including connections establishing)
tps = 2285.516762 (excluding connections establishing)

We then tried running the proxy using both TCP and Unix sockets, but the difference in average latencies for both of those proxy modes was negligible (~344ms vs ~339ms)
Command: pgbench -c 500 -j 2 -t 10000 postgres -h ~/cloudsql/[instance connection name] -U postgres

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 500
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 5000000/5000000
latency average = 344.194 ms
tps = 1452.669338 (including connections establishing)
tps = 1452.676626 (excluding connections establishing)

We ran the benchmarks again with 25, 50, and 100 clients instead of 500 and found that at 50 connections, the overhead latency is much, much smaller (~10ms instead of ~100ms). The numbers we were seeing were around ~16ms for SSL without the proxy and 28 ms for SSL with the proxy. At 25ms, we got a latency of ~9ms without the proxy and 13ms with the proxy. This makes sense given that the proxy would have more overheard per connection, given that there are actually 4x the number of connections involved (sending/receiving w/client-side proxy, and sending/receiving with server-side proxy) . At this point the Cloud SQL instance was at 100%, which shows the size of the instance is the bottle neck. It also makes sense that more connections would cause disproportional overhead given PG's connection pooling model (1 process per connection).

When it comes to optimizing, we compared the myCopy function in the proxy’s source code with the io.Copy function in the stdLib, and noticed the only real difference was the buffer size. We tried adjusting the default buffer size, but didn't see any noticeable performance improvements. We aren't sure what else we can do to reduce overhead (but are open to suggestions).

First, thank you for taking the time and diving into the issue. Really appreciated!
I also pointed GCP support to this issue, and they pointed me to #87 and #88 - which seems to suggest that measuring the actual application might be more important, as connection pooling could help here.
This is the results when testing our app using ab, 1000 requests, 100 concurent:

The table shows the difference between direct connection and connection via the proxy, and look like the mean and median are pretty similar (the numbers are milliseconds).
So this got me really confused TBH, as it might suggest we could ignore the slower performance reporter by pgbench.

Closing this issue, because the large overhead for connections only happens at very high connection volumes and shouldn't be an issue in typical use cases.


Should there be a design pattern for genetic disorders vs. unspecified?

I am looking at MONDO_0008698 (achalasia (disease)), which is a child of both idiopathic achalasia and inherited genetic disease.
I am looking because I am trying to understand representation of these CUI from UMLS:
[C0014848]  Esophageal Achalasia  (no genetic restriction)
[C2939435]  Hypertensive lower esophageal sphincter (no genetic restriction)
[C1860213]  Achalasia, familial esophageal  (also OMIM ACHALASIA, FAMILIAL ESOPHAGEAL [A11979735/OMIM/PT/200400])
Is there a rule to keep a distinction between heritable disorders and those for which the genetic basis is not part of the definition?
My questions:

should there be a distinction between Esophageal Achalasia  and Hypertensive lower esophageal sphincter?
should there be a distinction between what is currently represented as [C1860213] and [C0014848] /[C2939435]

Thanks

When known, we distinguish between heritable disorder and disorder for which the genetic basis is either unknown or non-existent. Heritable disorders get the axiom: ‘has modifier’ some inherited. Idiopathic disorders get the axiom: ‘has modifier’ some idiopathic.
With regards to this issue, it looks like the issues with the “achalasia” term have been resolved already:
The general concept “achalasia” (UMLS:C1321756, including both inherited and non-inherited) is the parent term for both inherited, 'achalasia, familial esophageal' (MONDO:0100457; UMLS:C1860213), and not inherited,  'idiopathic achalasia' (MONDO:0019635; UMLS:C0859976)
‘Esophageal Achalasia’ and ‘Hypertensive lower esophageal sphincter’ are not represented in Mondo as specific terms (their corresponding UMLS are currently “related” to the general parent term ‘achalasia’). Should we create new terms for these terms (note that they seem to be phenotypes and not diseases)? These would be children of the parent term “achalasia” or ‘idiopathic achalasia’ (MONDO:0019635; UMLS:C0859976), depending on whether the cause of the disease is known (though not genetic) or not, respectively.
@nicolevasilevsky : any thoughts regarding the creation of new terms?

I'm not opposed to adding the new terms if @maglott needs them and it is helpful for MedGen. I recommend that you go ahead. Thanks @sabrinatoro!

On behalf of MedGen, I don't think we need to add terms to Mondo if they are represented as phenotypes in HPO; but I do think it is important that if a familial form of a disease and a non-genetic form exist, they should not be lumped into a single concept (i.e. avoid equating an OMIM gene-specific record with a clinical presentation of unknown etiology). If that results in a pattern like:

Parent: Disease X

Child 1: Familial disease X (OMIM PS)

Grandchildren 1-N: Familial disease X, type 1 and  so on.. -N
Children 2-n: Acquired/idiopathic/post-infectious disease X (or whatever other etiologies/records exist)
Then there may be overlap in Mondo with HPO for the top "Parent" term, but I think Mondo has handled those rare instances pretty well and we don't really run into too many issues (except when there are shared preferred names on different concepts, but that's a different sort of problem for a different ticket)

Thanks @kanems!
Yes, we don't want to have a genetic form and idiopathic form of a disease lumped into the same concept.
@sabrinatoro I don't think there are any remaining action items, I will close.


FancyZones: Add tabs in zone with multiple windows

Description of the new feature / enhancement
If a single zone has many windows, it should (as option) show tabs to easily switch between them.
Scenario when this would be used?
e.g. when I'm working with multiple projects in multiple windows it will be very useful.
Supporting information
No response

Thank you for the proposal. I suppose it's a /dup of https://github.com/microsoft/PowerToys/issues/15921


404 at /get-docker/%20Docket

I found a broken link : https://docs.docker.com/get-docker/ Docket
I arrived on this page through: https://futureskillsnasscom.edcast.com/

Thanks for reporting; looks like someone made a typo on that website likely. The URL you're probably looking for is https://docs.docker.com/get-docker/
Given that this looks to be a broken link on a website we don't maintain (not in the docs itself), this is not something we can fix, so I'll close this ticket, but it may be worth reporting this issue to the owner of that website so that they can fix.


Freespace planner failed to plan a trajectory for parking if there is a vehicle very close to ego car

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I'm convinced that this is not my fault but a bug.

Description
When running the parking algorithm in the parking_lot area, there is an NPC vehicle very close behind the ego vehicle, which prevents the ego vehicle from running the freespace_planning. The NPC vehicle has not made contact with the ego vehicle and is not in the ego vehicle's driving path.
Expected behavior
Since the NPC vehicle does not affect the ego vehicle's parking, the freespace_planner should be able to plan a trajectory for parking.
Actual behavior
Unable to plan a trajectory for parking.
Steps to reproduce

Place ego vehicle in a parking_lot(should be big enough)
Spawn a NPC vehicle behind ego(should be close)
Set goal pose at a parking_space in the parking lot.

Versions

OS: ROS2
Autoware: humble-2023.08-cuda-amd64

Possible causes
Related code
// In file autoware.universe/planning/autoware_costmap_generator/nodes/autoware_costmap_generator/costmap_generator_node.cpp
// line 440
grid_map::Matrix CostmapGenerator::generateObjectsCostmap(
  const autoware_auto_perception_msgs::msg::PredictedObjects::ConstSharedPtr in_objects)
{
  const auto object_frame = in_objects->header.frame_id;
  const auto transformed_objects =
    transformObjects(tf_buffer_, in_objects, costmap_frame_, object_frame);

  grid_map::Matrix objects_costmap = objects2costmap_.makeCostmapFromObjects(
    costmap_, expand_polygon_size_, size_of_expansion_kernel_, transformed_objects);

  return objects_costmap;
}

// In file autoware.universe/planning/autoware_costmap_generator/nodes/autoware_costmap_generator/objects_to_costmap.cpp
// line 61
Eigen::MatrixXd ObjectsToCostmap::makeRectanglePoints(
  const autoware_auto_perception_msgs::msg::PredictedObject & in_object,
  const double expand_rectangle_size)
{
  double length = in_object.shape.dimensions.x + expand_rectangle_size;
  double width = in_object.shape.dimensions.y + expand_rectangle_size;
  Eigen::MatrixXd origin_points(NUMBER_OF_DIMENSIONS, NUMBER_OF_POINTS);
  origin_points << length / 2, length / 2, -length / 2, -length / 2, width / 2, -width / 2,
    -width / 2, width / 2;

  double yaw = tf2::getYaw(in_object.kinematics.initial_pose_with_covariance.pose.orientation);
  Eigen::MatrixXd rotation_matrix(NUMBER_OF_DIMENSIONS, NUMBER_OF_DIMENSIONS);
  rotation_matrix << std::cos(yaw), -std::sin(yaw), std::sin(yaw), std::cos(yaw);
  Eigen::MatrixXd rotated_points = rotation_matrix * origin_points;

  double dx = in_object.kinematics.initial_pose_with_covariance.pose.position.x;
  double dy = in_object.kinematics.initial_pose_with_covariance.pose.position.y;
  Eigen::MatrixXd transformed_points(NUMBER_OF_DIMENSIONS, NUMBER_OF_POINTS);
  Eigen::MatrixXd ones = Eigen::MatrixXd::Ones(1, NUMBER_OF_POINTS);
  transformed_points.row(0) = rotated_points.row(0) + dx * ones;
  transformed_points.row(1) = rotated_points.row(1) + dy * ones;

  return transformed_points;
}

When generating costmap from Objects, the value of parameters expand_polygon_size_ and grid_resolution are too large. I think that causes autoware to misjudge whether the starting point has an obstacle.
Additional context
https://github.com/user-attachments/assets/99624811-cf77-43ce-ba58-5b8b4a35909a
I initially discovered this bug when running with the Carla simulator. Here, I am using the planning-simulator to verify it.

@FZ-Broky
Thank you for your report!
With regard to the issue you raised, I would like to point out that several improvements have been done recently to improve the quality of free space planning, you can refer to this discussion.
I agree that the safety handling is still not ideal and needs improvement as you pointed out. But the changes applied include reducing the safety margins as you suggested.
If the result with latest autoware is still not good enough you can try reducing the following parameters to achieve the desired behavior:

expand_polygon_size
vehicle_shape_margin_m

@mkquda
Thank you for your reply! I will try on the latest version.

@FZ-Broky did using the latest version resolve your issues?

I completely agree with your viewpoint. Adjusting parameters may bring some short-term effects, but it cannot fundamentally solve the problem. As the ancient Chinese saying goes:
Treating symptoms without addressing the root cause，I think we should handle the coordination between speed planning and path planning well

@idorobotics I tried it on the latest version, and the issue still persists. I was wondering if we could remove the check for the start point.


Make str.length() return proper UTF8 string length

This fixes situations like this:
// tql2
from {}
len = "😅".length()

Before this change, len was 4. Now, it correctly is set to 1, counting the number of grapheme clusters.

I think the initial title was correct. That function counts UTF8 code-points instead of grapheme clusters. For example,
// tql2
from {}
len1 = "😅".length()
len2 = "é".length()
len3 = "é".length() // seemingly identical

I think the initial title was correct.

That's a good catch, thank you. So much for ChatGPT telling me Arrow was counting grapheme clusters, I should've verified.
We'll have to think about what we wanna do, if there's no good built-in support in Arrow then it's probably not worth spending the time, so I think we might want to stick with operating on code points for now.

I read a few articles and looked at what other languages or relevant standards like OCSF do here. My recommendation for now would be to add functions where the name explicitly indicates whether the functions returns number of bytes/number of codepoints/number of extended grapheme clusters. After getting some more experience with how these are used, we can think about what the "generic" length would be and potentially deprecate the long name for it.

Superseded by #4525.


Update toBe.lua

Fix: Allow :toBe(nil) to succeed when actual and expected are both nil

Updated the toBe matcher to handle nil values properly.
Added a special condition to check for both actual and expected being nil.

Hey there! Super cool that you're contributing with this PR. Much appreciated! Merging this right away :)


Non-zero extent of access_ptr is out range of the buffer

Currently, the extent is not exposed in access_ptr api of Buffer. But in this api implement, the total buffer size is used for extent argument of tvm_access_ptr.
As @tqchen mentioned in this discussion https://discuss.tvm.ai/t/what-is-the-precise-meaning-of-tvm-access-ptrs-extent/957 , if the offset is non-zero, then the end of access region offset+ extent will be out of range.
This is illustrated in this code sample.
m = tvm.var('m')
n = tvm.var('n')
Ab = tvm.decl_buffer((m, n), tvm.float32)
aptr = Ab.access_ptr("rw", offset=100)
assert tvm.ir_pass.Equal(aptr.args[3], m * n - 100)

closed by #1938


Issue with initializing KantoData

Continues from https://github.com/nilomr/pykanto/issues/40
@advaymishra2003 wrote:
... However, I am running into some other problems. I am setting up pykanto to process audio data but ran into an issue with initializing KantoData. Here’s the problem and what I tried:
Issue
When trying to initialize KantoData, I got this error:
AttributeError: 'Parameters' object has no attribute 'DATASET_ID'
It seems that KantoData expects DATASET_ID within the DIRS parameter. However, ProjDirs wasn’t accessible in the import, so I couldn’t set up DIRS properly. I attempted to workoaround this problem and I created a custom CustomDirs class to manually define the directory structure and provide DATASET_ID:
class CustomDirs:
def init(self, data_dir, output_dir):
self.DATA_DIR = data_dir
self.OUTPUT_DIR = output_dir
self.DATASET_ID = "your_dataset_id"
Then I used it as follows:
custom_dirs = CustomDirs(data_dir, output_dir)
dataset = KantoData(DIRS=custom_dirs, parameters=params)
Unfortunately, this workaround didn’t resolve the issue and led to the same error.
Question
Is there an updated way to set up DIRS with the current version of pykanto, or is there a recommended approach for defining the necessary attributes for KantoData initialization?
Thanks for your help!
Best,
Advay Mishra

Hi Advay,
Could you please provide a reproducible example, including traceback? In the meantime, the docs might help: setting up a project

@advaymishra2003 wrote:
Hi @nilomr ,
Thank you for getting back to me so quickly! Here's a reproducible example of the issue I encountered, along with the traceback.
import pykanto
from pykanto.dataset import KantoData
from pykanto.parameters import Parameters
Step 1: Define paths for data and output
data_dir = "wav_files"   # Directory containing audio (.wav) and JSON files
output_dir = "wav_files/output"  # Directory for output
Step 2: Initialize Parameters
params = Parameters(
data_dir=data_dir,
output_dir=output_dir,
lowcut=0,
highcut=7000,
window_size=0.03,
overlap=0.5,
threshold=0.1,
song_level=False
)
Step 3: Initialize the KantoData object
dataset = KantoData(DIRS=params, parameters=params)  # Using params for DIRS
Traceback
Here’s the traceback from the execution:
AttributeError: 'Parameters' object has no attribute 'DATASET_ID'
Traceback (most recent call last):
File "/opt/anaconda3/envs/pykanto_env_jupyter/lib/python3.9/site-packages/pykanto/dataset.py", line 110, in init
self.DATASET_ID = DIRS.DATASET_ID
AttributeError: 'Parameters' object has no attribute 'DATASET_ID'
Since I couldn't import ProjDirs (as it wasn’t accessible), I tried a workaround by defining a custom CustomDirs class with the required attributes (DATASET_ID, DATA_DIR, OUTPUT_DIR) and passing it as the DIRS argument. Unfortunately, this also resulted in the same error.
Let me know if you need any additional details or files to reproduce the issue.
Thanks again for your support!

Hi,
Please refer to the docs at https://nilomr.github.io/pykanto for examples of projects created using pykanto. As an example, here's one showing you how to segment vocalisations into units.
You can download these examples as Jupyter notebooks as well:

You can use your IDE or the pykanto docs to consult the documentation for each function. E.g., to see valid arguments and variable types that you can pass to the KantoData class, see pykanto.dataset.KantoData. Based on your example, it seems that you haven't created a ProjDirs object, which is the first argument that  KantoData expects. This is explained here: set-up-project-directories.
Lastly, am not sure what you mean by this: "I couldn't import ProjDirs (as it wasn’t accessible)". Tests for pykanto are running OK; from pykanto.utils.paths import ProjDirs works on my side.
Hope that helps!
Nilo


Changing array property in redux store doesn't change component's props

In redux-form I have an array property. Every time it's been changed store changes too, mapStateToProps recieves correct value, but componentWillReceiveProps recieve props only first 1-3 times (I don't know why amount is different), then nothing happens.
If I change another fields componentWillReceiveProps recieve correct values again.

Redux Form has its own repo
This sounds like a usage/support question which is better asked on Stack Overflow
You didn't provide any code to make it possible to diagnose your problem
Most likely you are mutating your array instead of returning a new one.

Redux Form has its own repo
Redux Form works here as expected
Most likely you are mutating your array instead of returning a new one.
Array is returning from 'redux-form/CHANGE' action, it doesn't mutate state

Sorry that I didn't provide code.
`import React, { Component } from 'react';
import { connect } from 'react-redux';
import { Field, reduxForm, formValueSelector } from 'redux-form';
import MenuItem from 'material-ui/MenuItem';
import Chip from 'material-ui/Chip/Chip';
import SelectField from '../../../ui/SelectField2';
const carBrands = ['BMW', 'Audi', 'Porsche', 'Maybach', 'Dodge'];
class List extends Component {
componentWillReceiveProps(nextProps) {
	console.log(nextProps.filter);
}

handleCustomDisplaySelections(values) {
	const chipStyle = { maxWidth: '100px', overflow: 'hidden', textOverflow: 'ellipsis' };
	if (values.length) {
		if (typeof values === 'string') {
			return <Chip labelStyle={chipStyle}>{values}</Chip>;
		}
		return (

{
				values.map((v, i) => {
					return <Chip labelStyle={chipStyle} key={i}>{v}</Chip>;
				})
			}

		);
	}
}

renderBrands() {
	return carBrands.map((item, key) => {
		return <MenuItem key={key} value={item} label={item}>{item}</MenuItem>
	});
}

render() {
	return (
		<form onSubmit={::this.props.handleSubmit}>

Brand

			<Field
				name="brands"
				hintText="Brand"
				id="brand"
				multiple
				component={SelectField}
				onChange={::this.props.change}
				displaySelectionsRenderer={::this.handleCustomDisplaySelections}
			>
				{::this.renderBrands()}
			</Field>

Active

			<Field
				name="is_active"
				id="is_active"
				hintText=""
				component={SelectField}
				onChange={::this.props.change}
			>
				<MenuItem value="both" primaryText="both" />
				<MenuItem value="true" primaryText="true" />
				<MenuItem value="false" primaryText="false" />
			</Field>
		</form>
	);
}

}
const selector = formValueSelector('services-filter');
function mapStateToProps(state) {
return {
filter: {
brands: selector(state, 'brands'),
is_active: selector(state, 'is_active'),
}
};
}
List = reduxForm({
form: 'services-filter'
})(List);
export default connect(mapStateToProps)(List);
`
Also video
https://youtu.be/3CXuVpPPn_w

I suggest trying https://github.com/leoasis/redux-immutable-state-invariant to verify that your state is not being mutated. Like 95% of issues with this symptom end up being mutated state.

You were right. Sorry for wasting your time


AHOYAPPS-435: Readme consistency

https://issues.corp.twilio.com/browse/AHOYAPPS-435

The really obvious changes to make are grouped in the first commit.
Not so obvious changes were done in separate commits so they are itemized and can be reverted if necessary. See commit message for description of what changed and why.
I will make comments for other changes I didn't think I should make myself without discussion.
I used shield style for CircleCI badge because that is what Ryan used for iOS readme and it seems more popular on GitHub.
I didn't add more screenshots besides the main app screenshot because nothing that would add value came to mind and they are difficult to maintain.

@Alton09 can you add a requirements section like other readmes?

Alright, I made the changes mentioned in my previous comment. @timrozum @aaalaniz

Alright, languages section removed ✅


Fix wrong guide information from apprtc params

Description
googNoiseSuppression is the correct one for Audio NS.
googNoiseReduction is for video frame denoise.
Purpose
This patch tells the right information about media constraints usage of "audio=googNoiseSuppression=false" or "video=googNoiseReduction=false".

there seems to be a considerable amount of confusion here:
https://source.chromium.org/search?q=googNoiseReduction&sq=
https://source.chromium.org/search?q=googNoiseSuppression&sq=
@guidou can you try to find someone who understands the difference?

cc @alebzk

Checked internally with audio people and this change looks good.

Thanks for the quick merge. Hope the live site of params.html can be updated soon as well, so that APPRTC users would not be misled any more.

@KaptenJansson can you deploy please?


[New Port Request] Flex and bison

Library name: Flex and bison
Library description: Compiler construction toolchain that builds LALR parsers with given context-free grammars. The parsers are generated in either C or C++.
Source repository URL: https://github.com/lexxmark/winflexbison
Project homepage (if different from the source repository):
Anything else that is useful to know when adding (such as optional features the library may have that should be included): Flex and bison comes with executables. flex and bison should become commands in the shell after installing

Duplicate to an existing issue that was closed before


ENUM types unsupported

I'm trying to read a file which contains ENUM types, however (e.g. parquet-tools schema shows something like required binary entryMethod (ENUM);) but parquetjs.ParquetReader.openFile() just throws an error:
Invalid ENUM value

... which means I can't use this package to read my files.
What are my options ?
Are there any plans to support this type ?
Thanks,

FYI I've posted a very similar problem to...
https://github.com/kbajalc/parquets/issues/3
because both projects seem to suffer the same problem.

Do you have a sample parquet file that fails?

@ZJONSSON, nothing that I can share right now ...
Sensitive data and all that.
For now I can share more output from parquet-tools - if that might help.
creator:                parquet-mr version 1.9.0 (build 38262e2c80015d0935dad20f8e18f2d6f9fbd03c)
<snip>
extra:                  writer.model.name = avro
<snip>
file schema:            <redacted>
--------------------------------------------------------------------------------
<redacted>
..entryMethod:          REQUIRED BINARY O:ENUM R:1 D:1
<snip>
row group 1:            RC:100000 TS:74883019 OFFSET:4
--------------------------------------------------------------------------------
<snip>
..entryMethod:           BINARY SNAPPY DO:0 FPO:39740111 SZ:227406/230832/1.02 VC:747359 ENC:RLE,PLAIN_DICTIONARY ST:[no stats for this column]
<snip>

I can try to find out how the file was created. Maybe see how to rustle one up.

Also, l FYI, when I look at the values for the ENUM columns, I can see that they are base 64 encoded strings.
Quote from https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#enum

ENUM
ENUM annotates the binary primitive type and indicates that the value was converted from an enumerated type in another data model (e.g. Thrift, Avro, Protobuf). Applications using a data model lacking a native enum type should interpret ENUM annotated field as a UTF-8 encoded string.
The sort order used for ENUM values is unsigned byte-wise comparison.

could anyone please let me know how to generate logical type "DECIMAL" ?


Add Treasure Island game

Hello folks,
If you want to contribute to this repository then go ahead you don't need to open an issue and ask me to assign you an issue. Also in Hacktoberfest issues don't play any major role unless the project is too high or well maintained with all rules and regulations.

hey @Arindam200 assign me i want to work on this issue

created PR please take a look @Arindam200


[Adaptive] Memory accessor shortcut does not resolve to an actual value

I tried @EntityName in my C# sample.  Here's what LUIS returns.
When I use @EntityName, I get the entire list back instead of the first value. JS implementation I believe points to the first value in the list - so effectively @EntityName is turn.entities.EntityName[0], #IntentName is turn.intents[0].key == "IntentName".
@boydc2014 @chrimc62 @Stevenic
{
  "recognizerResult": {
    "alteredText": null,
    "entities": {
      "$instance": {
        "todoTitle": [
          {
            "endIndex": 20,
            "startIndex": 4,
            "text": "finish this demo",
            "type": "todoTitle"
          }
        ]
      },
      "todoTitle": [
        "finish this demo"
      ]
    },
    "intents": {
      "AddToDoDialog": {
        "score": 0.999983847
      }
    },
    "text": "add finish this demo to the list"
  }
}

I believe that is on purpose.  If you want the first one you can explicitly refer to it, i.e. @foo[0].  Otherwise there is no way to pick up multiple entities of the same type.
From: vishwacsena<EMAIL_ADDRESS>Sent: Thursday, April 25, 2019 6:32 PM
To: Microsoft/botbuilder-dotnet<EMAIL_ADDRESS>Cc: Chris McConnell<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [Microsoft/botbuilder-dotnet] Memory accessor shortcut does not resolve to an actual value (#1798)
I tried @EntityName in my C# sample. Here's what LUIS returns.
When I use @EntityName, I get the entire list back instead of the first value. JS implementation I believe points to the first value in the list - so effectively @EntityName is turn.entities.EntityName[0], #IntentName is turn.intents[0].key == "IntentName".
@boydc2014https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fboydc2014&data=01|01|chrimc%40microsoft.com|abdfb71ba7aa40fd214508d6c9e6f2e3|72f988bf86f141af91ab2d7cd011db47|1&sdata=JhHP5KE%2B%2BfqQVshy82soWR24BQx2ouuXeE0OJhbAYAQ%3D&reserved=0 @chrimc62https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fchrimc62&data=01|01|chrimc%40microsoft.com|abdfb71ba7aa40fd214508d6c9e6f2e3|72f988bf86f141af91ab2d7cd011db47|1&sdata=3aw%2FRBHvxRsUOvn417AdpEhUXkVVYdAfj1Bh4kjN4BU%3D&reserved=0 @Stevenichttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FStevenic&data=01|01|chrimc%40microsoft.com|abdfb71ba7aa40fd214508d6c9e6f2e3|72f988bf86f141af91ab2d7cd011db47|1&sdata=KbBYZDnrYuAqGQYrhcYfFKLrwuNyH83AtVY4zqtJDcU%3D&reserved=0
{
"recognizerResult": {
"alteredText": null,

"entities": {

  "$instance": {

    "todoTitle": [

      {

        "endIndex": 20,

        "startIndex": 4,

        "text": "finish this demo",

        "type": "todoTitle"

      }

    ]

  },

  "todoTitle": [

    "finish this demo"

  ]

},

"intents": {

  "AddToDoDialog": {

    "score": 0.999983847

  }

},

"text": "add finish this demo to the list"

}
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fbotbuilder-dotnet%2Fissues%2F1798&data=01|01|chrimc%40microsoft.com|abdfb71ba7aa40fd214508d6c9e6f2e3|72f988bf86f141af91ab2d7cd011db47|1&sdata=K7SEJQgbA6jXG5pnby0Qmg2kfav65eGE3Pss2RX3JLc%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEGSCMNHZV6AJVZJVGNJDKTPSJLQHANCNFSM4HIRZUNQ&data=01|01|chrimc%40microsoft.com|abdfb71ba7aa40fd214508d6c9e6f2e3|72f988bf86f141af91ab2d7cd011db47|1&sdata=RlmZyrwjt96r%2BJK2ASZsgvAIy%2FCB2DwyNUwi0OfeCZU%3D&reserved=0.

close due to dup


Update website for datajam prize money

We're going to offer a $1000 first place prize and a $500 best use of ESRI technology prize at Data Jam.

How/where do you want me to put this information?

I wouldn't do this as citycamp. the big we are trying to is get out of this
business and let Zach cover it
Chris Mathews
On Mon, Sep 19, 2016 6:38 AM, lenaecb<EMAIL_ADDRESS>wrote:
How/where do you want me to put this information?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.


Error upgrading from 1.1.3 - Unable to Read Previously Saved State for UpgradeResourceState

Hi!
We're trying to upgrade from 1.1.3 to 1.3.0 and getting the following in the plan:
╷
│ Error: Unable to Read Previously Saved State for UpgradeResourceState
│
│   with opslevel_check_service_property.language_defined,
│   on basic_checks.tf line 33, in resource "opslevel_check_service_property" "language_defined":
│   33: resource "opslevel_check_service_property" "language_defined" {
│
│ There was an error reading the saved resource state using the prior
│ resource schema defined for version 0 upgrade.
│
│ Please report this to the provider developer:
│
│ AttributeName("predicate"): invalid JSON, expected "[", got "{"
╵

The check is defined as follows:
resource "opslevel_check_service_property" "language_defined" {
  name     = "Service Language Defined"
  enabled  = true
  category = data.opslevel_rubric_category.ownership.id
  level    = data.opslevel_rubric_level.meets_expectations.id
  property = "language"
  predicate = {
    type = "exists"
  }
  filter = opslevel_filter.not_third_party_services.id
}

Hey @lboynton - I think you are running into this problem documented in our migration guide
If you have created checks in 1.0.x or 1.1.x you will first need to upgrade to 1.1.4 before jumping to >= 1.2.5

https://github.com/OpsLevel/terraform-provider-opslevel/blob/main/MIGRATION.md

Oh ok thanks, will give that a go. I thought it might be something simple!

Confirming that after doing a plan/apply with 1.1.4 this error has now gone, thanks!

Thanks for confirming @lboynton - we appreciate you as opslevel customers!


Observing Poor Performance on Our Independent Models

Hi Ethan,
My name is Chethan. I've created a version of your CLMBR pipeline, but am observing poor performance on my independent models (i.e. logistic regression and random forest). I’ve spent time figuring out why this is happening that includes…1) re-checking my coding, the paper, and repo to make sure the logic is sound, 2) exploring different hyper-parameter values for CLMBR, and 3) explored hyper-parameter tuning of the independent models. All three approaches have led to still poor performance. I'm wondering if you'd be available for a 1 hour meeting this or next week?  Let me know your availability.

Hi Chethan,
Sorry to hear that you have been getting poor performance! I'd be glad to talk to you to work out what is going on. Sorry for the late reply, I accidentally had email notifications for github turned off.
I'll be free during the following times next week:

Not a problem, I understand that. how about this friday (12/17) from 11am-12pm? I can send a google meeting invite to your stanford email address. I'm using claims data that contains CCS, CPT, and HCPCS codes. I'd like to walk you through my implementation to get your thoughts. I agree is most likely how the data was extracted and even the duration of an encounter.

I'm in the Pacific Time, so no issue. I'll send an invite over.

I should have mentioned this from the beginning, the nature of our data prevented us from using your extraction process, so we created our own code to process the data to then get fed into our model. We believe a possible issue might be how we represent an encounter. We consider a day an encounter for each patient, but speaking with your professor we learned that the notion of a "day", or the encounter duration that you use matters a lot.

Why don't you send me a copy of your code before we chat? That way I could take a look and be better prepared to ask the right questions when we talk?

Sure sounds good. I'll email you our data preparation and modeling code over to you. The former will require a bit of cleaning and commenting, so I'll send over the modeling code in the meantime.


⚠ File access (beta) has degraded performance

In 6974a4e, File access (beta) (https://beta.fyle.uk/file/fish) experienced degraded performance:

HTTP code: 200
Response time: 9468 ms

Resolved: File access (beta) performance has improved in f4a1571.


Primeng's styles config is not working

Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Versions.

angular-cli 1.0.2
Windows 7/10
macOS Sierra
Repro steps.

When intergrating Primeng, the primeng's styles are not included.
You can easily reproduce it by building the primeng's official example.
My styles config in .angular-cli.json is:
"styles": [
        "styles.scss",
        "../node_modules/primeng/resources/primeng.min.css",
        "../node_modules/primeng/resources/themes/omega/theme.css",
        "../node_modules/font-awesome/css/font-awesome.min.css"
      ],

The log given by the failure.

Desired functionality.

I don't know it's the problem of angular-cli's build script or primeng's. Please indicate a way to solve this. Thanks.
Mention any other details that might be useful.

I'm sorry, but this issue is not caused by Angular CLI. Please contact the author(s) of the primefaces/primeng-quickstart-cli project or file an issue on their issue tracker.


Refreshed icon set

Creted new icons for uniformity, brand recognition and coolness factor,
based on original logo by Asysr.
Removed old pixmap files, .icns file, and overview.png_old.

Nice attention to detail, very good :)

Sweet :)

FYI, this change broke development builds on OSX.
OSX requires an icon in icns format and the gridcoinresearch.pro file points to a now deleted icon file.
I've made a PR with the fix here #251


✨ Shift-Click to Make Text Selection

Users should be able to make a selection by holding down the Shift key and clicking anywhere in the document to highlight text from where the cursor was to where they just clicked.

This now works, but only half-way. It only works on the first click. When I click a subsequent time further down, it does not update the selection to include the additional characters/lines.

This works now, closing.


x/tools/go/loader: tests are too brittle

The x/tools/go/loader tests have a history of breaking whenever something in the standard library
changes.

I think it's time to rethink those tests to not be a maintenance burden.
(Latest example: https://go-review.googlesource.com/c/37148/)
/cc @ianlancetaylor

Another instance is #38318.

Given that golang.org/x/tools/go/loader is now deprecated, I think it's reasonable to say that no one will probably have the time/desire to invest into fixing this issue. In that case, I think we can either continue to fix breakages like https://github.com/golang/go/issues/38318 as they come up (and make sure that they get caught sooner rather than later), or decide that these tests aren't worth it and disable them. Given the discussion on https://golang.org/cl/227772, I would say that we do consider these tests worthwhile, in which case I propose we close this issue and fix breakages as they happen.


Add Deno support in Relay Plugin

The relay plugin currently does not support Deno due to the usage of the Buffer global.
see example https://github.com/hayes/pothos/blob/8ca9b66cf7c9bb63c5d100757ffad65c17861f9d/packages/deno/packages/plugin-relay/utils/connections.ts#L100
Fortunately Buffer has an implementation in the deno std lib https://deno.land/std@0.152.0/node/buffer.ts?s=Buffer
I'm happy to open a PR to dynamically import the class if it does not exist. I think a static import would be better but I took a look at the build script and I don't see a clean way of doing it

Whoops, thats a good point.   It's just used for base64 encoding, should be easy to make this work in any environment.  I'll have to check if it's supported in all node versions, but just using atob and btoa would probably work well, if prefer to not use deno stdlib unless it's needed so this also works in browsers

I think if atob and btoa are available then using that is the easiest, although Deno std lib should work in browser provided that its transpiled from typescript first. Could do somthing like
const b64 = typeof btoa === "undefined" ? (s) => Buffer.from(cursor, "base64").toString() : btoa

published a new version that should fix this


different session configs with the same path only execute for the first one

What happened?
When I config some sessions with the same path directory:
[[session]]
name = "update nvim"
path = "/tmp"
startup_command = "brew upgrade neovim --fetch-HEAD && exit"

[[session]]
name = "update all"
path = "/tmp"
startup_command = "brew update && brew upgrade && exit"

I expected update nvim and update all to run their own session but only update nvim was executed regardless of session!
Version
1.1.0
Relevant log output
No response
Reviewed

[X] Reviewed existing issues and made sure this bug has not already been submitted.
[X] I have read the documentation and understand how sesh works before submitting this bug

correct me if I'm wrong, but I think we should return the found session config command here and pass it to connect function.
https://github.com/joshmedeski/sesh/blob/fe6e6a65d8cb9dbc2f0ab5abdbe15351c5504d96/session/determine.go#L16C4-L20C5

I am actively reworking the connect logic to handle this situation, thanks for flagging this. The first version of sesh was very much dependent on determining the path first, so this new config pattern breaks that pattern.
I'll be sure to mark this issue closed when I complete the work.

I think I have just realised my setup is suffering from this issue too. My sesh.toml has two sessions configured to switch between qlt and tst cloud environments for which Infra-as-Code is maintained in the common Git repository:
[[session]]
name = "qlt"
path = "~/azure-hosting"
startup_script = "~/.config/sesh/scripts/azure-hosting-qlt.sh"

[[session]]
name = "tst"
path = "~/azure-hosting"
startup_script = "~/.config/sesh/scripts/azure-hosting-tst.sh"

The sesh connect tst does not load tst environment session, but seems to load the first one in the config, that is sesh connect qlt.

This should be fixed in sesh v2, There's a beta release if you want to test it. The full release is coming in a couple of weeks!

I'm happy to confirm the issue I described in https://github.com/joshmedeski/sesh/issues/97#issuecomment-2280724523 has now been fixed - I just had to s/startup_script/startup_command/. Thank you @joshmedeski for v2.0!

The startup command will only match by name (not path) so the issue is fixed with the v2 launch!

@joshmedeski Late August I confirmed the fix https://github.com/joshmedeski/sesh/issues/97#issuecomment-2307078446 works for me. I've just upgraded to latest sesh version dev and I'm observing the problem is back and this sesh.toml I've been using is no longer switching to azure-hosting-tst but always loads and connects azure-hosting-prd
[[session]]
name = "azure-hosting-prd"
path = "~/azure-hosting"
startup_command = "~/.config/sesh/scripts/azure-hosting-prd.sh"

[[session]]
name = "azure-hosting-qlt"
path = "~/azure-hosting"
startup_command = "~/.config/sesh/scripts/azure-hosting-qlt.sh"

[[session]]
name = "azure-hosting-tst"
path = "~/azure-hosting"
startup_command = "~/.config/sesh/scripts/azure-hosting-tst.sh"

Interestingly, if I downgrade from sesh version dev to sesh version 2.4.0 then everything works for me again.
IMHO, this issue needs to be re-opened.

Thanks for the update, I can do some debugging and see if I can recreate the issue.
Right now, dev doesn't introduce any new logic from 2.4.0, so I'm wondering why you're getting different results.

To clarify, session config matching only happens by name now (not path), so whatever the bug is will be related to something else.

@joshmedeski

I can do some debugging and see if I can recreate the issue. (...) dev doesn't introduce any new logic from 2.4.0, so I'm wondering why you're getting different results.

I wish I could help. I can only observe difference in behaviour between the 2.4.0 and current dev.
Perhaps this will help in testing:

Here is very simplified version of my sesh configuration and scripts, https://github.com/mloskot/dotfiles/tree/ml/sesh-issue-97/sesh

Here is screencast where I replicate my observation of 2.4.0 vs dev:
Screencast from 2024-10-05 12-44-40.webm

Here are commands from the screencast which I run to replicate the observations:
go install github.com/joshmedeski/sesh@latest

sesh --version

sesh connect azure-hosting-prd   # prd is loaded
# tmux kill-server to start fresh
sesh connect azure-hosting-tst   # tst is not loaded, but prd is loaded again

curl -o sesh.tar.gz -L https://github.com/joshmedeski/sesh/releases/download/v2.4.0/sesh_Linux_x86_64.tar.gz && \
tar -xf sesh.tar.gz && \
mv sesh $HOME/go/bin

sesh --version

sesh connect azure-hosting-prd   # prd is loaded
# tmux kill-server to start fresh
sesh connect azure-hosting-tst   # tst is loaded

Got it, thanks for the detailed feedback. I can take a closer look at the logic that determines which startup script to run, there is probably a bug in that specific area.

@joshmedeski  Thank you for looking into the issue.

For now, can you please stick to using the latest release?

Yes, of course. I'm happy to

And which OS are you using?

I'm on Ubuntu 24.04

Perhaps I can add support for your package manager to make it easier.

It's always nice to have .deb -s downloadable for software releases on GitHub, but if it is any burden for you then there may be no point adding it as the tandem of curl and tar do the job well.
Thanks!

Thanks,  I'm going to close this for now.
p.s. I worked on deb support before (https://github.com/joshmedeski/sesh/issues/68) and ran into some issues, so I'll revisit this again soon.


Add a token that's required to be invited.

Meant to be distributed in real life, or via email, or whatever other form you want. For those times when you don't want to invite everyone, and don't have a list of email addresses.
Totally understand if you don't want this, but I thought I'd offer it up.

It can be useful. Can you add some description in README?

I'll add something to the readme now.

@lfn3 Sorry, I was super busy. I tested it but there are a tokenRequired is not defined error.
Because I think this is good feature and I am super late, I will merge it and fix it above your commit.


ADR: Bitnami Supabase deprecation continuation

Describe what should be investigated or refactored
Supabase has been deprecated by Bitnami, so there's a need to determine a path forward. This ADR should determine what path we're going to take.

Switch to supabase-community and maintain custom configs
Maintain the bitnami charts/images
Move off of Supabase

ADR Supplement: Supabase Deployment Options
Primary Focus
This ADR supplement evaluates the maintainability and initial effort required to set up and maintain various Supabase deployment options, including supabase-community, maintaining the deprecated supabase/bitnami charts + images, and assessing alternatives listed on the official Supabase website.
Deployment Options
1. Supabase-Community (https://github.com/supabase-community/supabase-kubernetes)

Effort

Approximately two weeks for one person, primarily dealing with Helm charts and ensuring clean integration with LeapfrogAI.
Parallelizing configuration work by service (realtime, Kong, REST, storage, etc.) across multiple contributors could potentially reduce this timeframe.

Additional Considerations

Database migration testing may require extra time. Ideally, this would involve swapping out the PVC, but additional work may be necessary if the new Postgres container doesn't seamlessly adopt existing database files, variables, and configurations.
Requires comparing the new supabase-community values file and templates with our existing values file and the previous Bitnami values file and templates.

Advantages

Actively maintained
Good Kubernetes deployment base and images
Supports all Supabase features
Potential to contribute changes back to the upstream repository

Initial Integration Notes

Initial configurations available on the branch 968-adr-bitnami-supabase-deprecation-continuation
Working Docker containers compatible with a uds-core cluster
Functional Kubernetes deployment with all Supabase components running and networked
Supabase accessible via "https://supabase-kong.uds.dev" with working dashboard, database, and realtime features
DB migration container runs sucessfully
Operational Zarf package
Not yet ready for LeapfrogAI integration; requires at minimum adjustments to secrets management
Configs not production-ready; includes hardcoded credentials in charts and default values needs additional evaluation for security and sensibility
Requires additional templates to reduce manual configuration
Some configurations and templates may be adaptable from the Bitnami chart and alternative setups

Required Improvements

Implement JWT job/secrets
Enhance database persistence to prevent accidental deletion
Secure Postgres credentials
Align declarative config names
Test and potentially modify Supabase Realtime changes for Kong
Adjust max file size for storage
Ensure secrets availability so that other components can complete startup and migrations, namely the LeapfrogAI UI and API
Address any issues with pgvector and dashboard configurations
Resolve any read/write issues due to dockerfile user permissions
Configure gotrue site URL for auth and Keycloak integration

Maintenance Requirements

Maintain two small Dockerfiles primarily to resolve permissions issues due to assumed root user
Additional Dockerfiles may be necessary for extended feature support
Ongoing maintenance of templates and values file for use with the base supabase-community Helm chart

Resources

https://github.com/supabase-community/supabase-kubernetes/tree/main
https://github.com/defenseunicorns/leapfrogai/compare/main...968-adr-bitnami-supabase-deprecation-continuation

2. Alternative Supabase Versions (https://supabase.com/docs/guides/self-hosting)
2.1 Maintaining Fork of Deprecated Supabase Bitnami Images/Charts

Effort

Fastest setup (estimated 1-2 days) due to minimal changes in chart configurations and images. This time would primarily involve pushing updated images, adjusting configurations to reflect new image locations, and establishing a new repository for ongoing maintenance.

Advantages

Fastest setup option
No database migration required
Useful as a reference for supabase-community configuration

Disadvantages

Removed from Bitnami repo; full maintenance responsibility falls on our team
3 external dependencies from Bitnami

Maintenance Requirements

Requires maintaining 5 custom images and 1 chart with 7 templates and in-depth understanding of Supabase.
Potentially maintaining the Kong chart if Bitnami decides to remove it.

Resources

https://github.com/bitnami/charts/commit/4b5d3cc2409889d8dfbef0844cfbfdf5879223f6
https://github.com/bitnami/containers/commit/b373dabb821e7c564f10af2336afbda4ff477765
https://github.com/defenseunicorns/leapfrogai/tree/main/packages/supabase

2.2 Stackgres Supabase Deployment

Effort

Similar deployment time to supabase-community, but introduces an additional dependency that may add a couple days.

Advantages

Useful as a reference for supabase-community configuration

Disadvantages

Built on top of supabase-community
Adds an opinionated Kubernetes Postgres deployment and operator
Requires same setup time as supabase-community plus additional dependency
Not actively maintained; serves as an example deployment

Maintenance Requirements

Same as supabase-community in addition to stackgres configuration

Resources

https://stackgres.io/blog/running-supabase-on-top-of-stackgres/
https://github.com/ongres/apps-on-stackgres/tree/main/examples/supabase

2.3 DigitalOcean Deployment

Effort

Not applicable for direct deployment within our environment

Advantages

Useful as a reference for environment variables

Disadvantages

Combines Stackgres and Terraform
Year-old deployment example
Hybrid Kubernetes-Docker deployment
Not suitable for our purposes (SAAS deployment)

Maintenance Requirements

Not applicable for direct deployment within our environment

Resources

https://github.com/digitalocean/supabase-on-do
https://github.com/digitalocean/supabase-on-do/blob/main/packer/supabase/docker-compose.yml

2.4 AWS Deployment

Effort

Not applicable for direct deployment within our environment

Advantages

Useful as a reference for environment variables

Disadvantages

AWS-focused deployment using CloudFormation
No Kubernetes configs
Not suitable for our purposes (SAAS deployment)

Maintenance Requirements

Not applicable for direct deployment within our environment

Resources

https://github.com/supabase-community/supabase-on-aws/tree/main

2.5 Postgres AIO Docker Container

Effort

Estimated 2 weeks to 1 month for one person to create a new Helm chart and associated templates.

Advantages

Useful as a reference for supabase-community configuration
Simplest implementation in terms of number of pods and configuration between services, may integrate with istio without modification

Disadvantages

Not suitable for production use in its current state, built for Postgres testing
Lacks charts, requiring full chart and template maintenance
Unclear feature support
Infrequently maintained

Maintenance Requirements

Requires an in-depth understanding of Supabase to maintain a single complex dockerfile, helm chart, all templates, and values.

Resources

https://github.com/supabase/postgres/tree/develop/docker/all-in-one

2.6 Supabase Terraform

Effort

Not applicable for direct deployment within our environment

Advantages

None

Disadvantages

Similar to DigitalOcean deployment but unmaintained and focused on terraform

Maintenance Requirements

Not applicable for direct deployment within our environment

Resources

https://github.com/supabase-community/supabase-terraform

2.7 Supabase Traefik

Effort

Not applicable for direct deployment within our environment

Advantages

None

Disadvantages

Docker Compose-centric deployment
Unmaintained
Likely superseded by updates from supabase-community and official Supabase Docker images

Maintenance Requirements

Not applicable for direct deployment within our environment

Resources

https://github.com/supabase-community/supabase-traefik

Recommendation
This analysis suggests that supabase-community offers the most promising long-term solution due to its active maintenance, comprehensive feature support, and potential for community contributions. While maintaining a fork of the deprecated Bitnami chart offers the fastest initial setup, it carries a significant, ongoing maintenance burden. Other alternatives are either unsuitable for our environment or lack the necessary features and maintenance support.


Update UniqueValidator.php

Q
A

Is bugfix?
✔

New feature?
❌

Breaks BC?
✔/❌

Fixed issues
#19407

I don't see how your solution is addressing the problem in any way.

As I explained in details already in my fix attempt for this issue I stand behind my version.


Passing input type List to field resolves to null

the following used to work on rc4, but fails on rc5
query { a:(foo:[1,2,3]) }

argument foo in resolver is null

this was broken by commit  (https://github.com/ChilliCream/hotchocolate/commit/cf6aab0ccac0f841afd7d6dd7add96a582012bc5)


[pt-br] Add docs/reference/issues-security/security.md

Translate docs to pt-br
docs/reference/issue-security/security.md
Related issues
#13939

/assign stormqueen1990

Hi @italofernandes13!
I'm a bit busy this week but will come back to review this PR as soon as possible!
/cc

/retitle [pt-br] Add docs/reference/issues-security/security.md

/cc @rikatz
Could you PTAL?

/lgtm

/label tide/merge-method-squash

/hold
For checking aliases in the front matter.

/lgtm

/cc @devlware
Could you please provide a second review?

Hi @italofernandes13, If you need help with the proposed changes, please let us know.

Hello there, @italofernandes13! 👋
Thank you very much for your work on this PR. However, it has been open and reviewed for a while with no action taken. For this reason, I will be closing this PR. If you wish to continue your work on this, please feel free to reopen it.
Should you have any questions, please don't hesitate to reach out to the #kubernetes-docs-pt channel in Kubernetes Slack.

Olá, @italofernandes13! 👋
Obrigada pelo teu trabalho neste PR. Devido à inatividade após a revisão, irei fechá-lo. No entanto, caso desejes continuar teu trabalho, podes reabrir o PR a qualquer momento.
Se tiveres dúvidas, não hesita em contatar-nos através do canal #kubernetes-docs-pt no Slack do Kubernetes.
/close


How to make custom dataset only with Lidar information？

Thanks for your great work.
I just have labeled pointcloud data ,can I train  lidar-only bev model?with custom dataset?
best wishes!

Thank you for your interest in our project. This repository is no longer actively maintained, so we will be closing this issue. Please refer to the amazing implementation at MMDetection3D. Thank you again!


Unexpected error at login

After I tried to login with my credentials I've got the following error message. But password and e-mail was correct. I've also changed the password to another one which has no special chars. Without success.

Please check the home assistant log and append the error message here.

Sorry, here it is. We have an Dachs with 5KW
2024-09-11 07:43:34.822 DEBUG (MainThread) [custom_components.senertec.config_flow] Trying to connect to senertec during setup
2024-09-11 07:43:34.822 INFO (SyncWorker_16) [senertec.client] Logging in..
2024-09-11 07:43:35.197 INFO (SyncWorker_16) [senertec.client] Login was successful.
2024-09-11 07:43:35.198 INFO (SyncWorker_2) [senertec.client] Initializing senertec platform...
2024-09-11 07:43:35.470 ERROR (MainThread) [custom_components.senertec.config_flow] Unexpected exception
Traceback (most recent call last):
  File "/config/custom_components/senertec/config_flow.py", line 78, in async_step_user
    info = await validate_connection(self.hass, user_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/senertec/config_flow.py", line 45, in validate_connection
    await hass.async_add_executor_job(client.connectUnit, units[0].serial)
                                                          ~~~~~^^^
IndexError: list index out of range
2024-09-11 07:43:35.498 INFO (senertec-websocket) [websocket] Websocket connected
2024-09-11 07:43:35.498 INFO (senertec-websocket) [senertec.client] Connected to Senertec websocket

The login works as I can see. But there are no heating units returned.
Seems like the same strange thing as discussed here
Unfortunately I have no solution for it because I don't own these heating units and can't test furthermore.

If you have time for debugging with the py-senertec library, I can create a test release with more debugging information.
Then we can take a look deeper what's the problem is.

Maybe the reason for that is bc our Dachs is marked as offline in the Dachs Portal. We are in contact with our service provider about this situation.

The reason for this error was, that the unit was not implemented completly. Now after our service provider configured this in dachsportal, it works fine.


Conflict resolution controls doesn't appear with 1.4.x

I was experiencing problem since a couple days with this plugin. Detection works. It renders the listing correctly and a clicked file appears. It's then that I don't see the resolution overlay. It's just the raw un-merged file.
I tried disabling every other plugin, then restarted Atom, same problem.
I reverted to 1.3.7 and it worked.
I'm on windows with Atom version 1.7.1

Same here.
It lists the conflict, but clicking on a file does not result in the highlighted conflict areas to work on.
No idea how long it's been in this state, as I haven't used the package in a while.

This is likely a result of the ConflictParser changes that were made recently to recognize diff3 output. I think we're outgrowing the hacky mechanism that I'd been using to parse conflict markers. I've got a branch locally where I've been refactoring that into something more maintainable; hopefully I'll catch the problem here along the way.
A few questions to clarify what's going on:

Does it occur with a specific project or file, or have you seen it happen in different places?
What platform are you on? If you're on Windows, do you have \r\n line endings?
If you can, can you provide a sample of a file (sanitized) with the conflict markers that isn't getting the overlay correctly, especially if it does work on other files?

Happened to me with two different projects, one javascript, one custom file type.
Windows 10.
Unfortunately, I have already gone through corrections. Next time I encounter this, I'll save a copy and post it.

Sounds good. I'm most suspicious about those \r\n endings right now; the local smoke tests I do don't cover those yet.

Well I can tell you that my files are both written and committed with Windows line endings, e.g. something like this:

But that's always been the case and it used to work when I first tried this package.

I just tried to use this module for the first time with a merge conflict yesterday. It detected the files that had conflicts, but when I opened them it simply displayed the file with the normal git markers in it. I am running Atom 1.8.0-b2 on Windows 10 but the file in question has on LF line endings (everything we do is LF-only). I haven't been able to test it more than this one particular merge conflict which was only a single file. I found this existing reported issue when researching the problem. Unfortunately, though, I didn't keep a sample of the conflict marker area. Anyhow, I just wanted to chime in and say at least for me, I had the problem with just LF endings on a Windows platform.

But that's always been the case and it used to work when I first tried this package.

Yeah, a recent change that modified that regular expression which is a likely culprit; regular expressions are notoriously tricky and this one's growing in complexity.

I am running Atom 1.8.0-b2 on Windows 10 but the file in question has on LF line endings (everything we do is LF-only).

Oh weird. I'll have to do some testing downstairs on the Windows machine to see if I can replicate that one. Thanks for the data point :100:

I'm on windows and I know for sure that I'm using LF line endings on my projects as I'm enforcing it through git, and Atom confirms that. With version 1.3.7, both line endings work.
I encountered the problem with different projects and different file types, I remember mostly js.
I made a test:

Updated merge-conflicts to 1.4.1
created a new text file. LF endings, on windows 10.
I created a conflict

Aaaaannnd it worked well...
So I made a second test, this time with a js file, and I got the problem.

I also tried disabling all other installed community packages and the problem was still present.

I just tried with 1.4.2 (uninstalled and installed a fresh merge-conflict package). Atom 1.8.0-beta2. Unfortunately, it is still failing to detect the markers or something. This test file is UTF-8, LF endings, and Ruby language.

Possible duplicate of #223

I just tried with 1.4.2 (uninstalled and installed a fresh merge-conflict package). Atom 1.8.0-beta2. Unfortunately, it is still failing to detect the markers or something.

Nuts, I was hoping that #229 would take care of this without needing more complicated diagnostics :smile:
I just gave this a shot on my Windows 10 machine downstairs and... couldn't replicate it. I'm not using Atom Beta there yet, though. I'll give that a try later and see if there's a regression there.


Project timestamp to date

Hello,
I want to write following query in laravel.
db.foo.aggregate({
$project: {
date: { $add: [new ISODate(0), "$time"] }
}
})
Please tell me there is any way to do this. I have tried following code but got error.
Foo::raw()->aggregate(['$project' => ['date' => [ '$add' => ['new ISODate(0)', '$time']]]]);
Error:
exception: $add only supports numeric or date types, not String
Thanks

change 'new ISODate(0)'
to new \MongoDate()