Loading of all assemblies when loading an image

I just noticed this when running the app in debug mode:
https://github.com/JimBobSquarePants/ImageProcessor/blob/master/src/ImageProcessor/Configuration/ImageProcessorBootstrapper.cs#L80
We have about 100 DLLs in there. Some of them are C++ (non-.Net) DLLs, so it throws a bunch of errors. Some of them weigh more than 10Mb. This doesn't seem very optimized... :/
I understand that you want to discover the available formats without explicitly linking or listing them, but is there a way to filter the loaded DLLs ?

Hmmm.... I guess we could optimize by filtering by some sort of naming convention. However, there isn't a convention and I don't know what people have called their stuff in the wild.

Maybe some sort of config (in a new ImageProcessor.config file) to disable auto-loading of other DLLs (which behaves like today by default), and another config with a regexp to filter the file names ?
I don't know about slowdowns, because it's always been this way (I just added the PDBs so I assume it used to do that without me knowing about it). I must assume loading 10mb files from the disk must slow down the processing in some way, not necessarily in a really significant way for my app, but maybe for others.

It'll only happen on first initialisation and is cached after that and shouldn't affect further processing.
I loath the idea of adding config files to ImageProcessor, parts of it's charm is its install and go functionality.

Yes but you can use optional config files. Meaning that if the config file is not found, or if the value is not found in the config file, it has a default value, which behaves like today.
It would still work out-of-the-box, but could give users options to optimize some areas.

Feel free to have a tinker at that then, I'm a bit swamped at the mo. Remember.. v2 branch :wink:

Ok, not sure I'll have time just now either, but I'll think about it :smile:

Fixed in 2.2.1 I've replaced the type loader with a new optimized version that filters out a large number of known assemblies when looking for plugins..

I am experiencing a similar situation, when in use with AutoMapper. ImageProcessor for some reason tries to load AutoMapper, because one of the DLLs is referencing AutoMapper. However AutoMapper.dll is not even present in the bin folder, so don't know why ImageProcessor even tries to load that.
I was thinking, maybe it is possible to turn off plugin search via a config file?

Can you provide some sort of prove for this please?

Basically, I have two environments:

Mine
Client's.

I am not experiencing problems, my client does.
I have an App, Library, AutoMapper.
App references Library, which one method uses AutoMapper. However, App never calls that method, so App works on my PC without AutoMapper. Nevertheless, my client gets an exception, which StackTrace is below.
I use "<IP_ADDRESS>" version of ImageProcessor. Both, client and me use .NET Framework 4.5.
System.AggregateException: One or more errors occurred. ---> System.IO.FileLoadException: Could not load file or assembly 'AutoMapper, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=be96cd2c38ef1005' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515) ---> System.NotSupportedException: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
   --- End of inner exception stack trace ---
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at ImageProcessor.Common.Helpers.TypeFinder.GetAllAssemblies()
   at ImageProcessor.Common.Helpers.TypeFinder.GetFilteredAssemblies(IEnumerable`1 excludeFromResults, String[] exclusionFilter)
   at ImageProcessor.Common.Helpers.TypeFinder.GetAssembliesWithKnownExclusions(IEnumerable`1 excludeFromResults)
   at ImageProcessor.Configuration.ImageProcessorBootstrapper.LoadSupportedImageFormats()
   at ImageProcessor.Configuration.ImageProcessorBootstrapper..ctor()
   at ImageProcessor.Configuration.ImageProcessorBootstrapper.<.cctor>b__8()
   at System.Lazy`1.CreateValue()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.get_Value()
   at ImageProcessor.Imaging.Formats.FormatUtilities.GetFormat(Stream stream)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at MyClass.ImageManipulation.Rotate(Byte[] imageData, Single degrees) in C:\Source\ImageManipulation.cs:line 31

Thanks for that. I appreciate it.
ImageProcessor will load a reference to the assembly but not try to parse the assembly to find any available types if it is in the excluded list.
That error is caused either by an Automapper.dll in the bin folder that is blocked as it was downloaded from the internet....
http://blogs.msdn.com/b/drew/archive/2009/12/24/xunit-and-td-net-fixing-the-attempt-was-made-to-load-an-assembly-from-a-network-location-problem.aspx
.... or because the assembly is somehow references from a remote location or a Virtual PC application when you try to load a file from linked folders on the hosting computer.
https://msdn.microsoft.com/en-us/library/dd409252(VS.100).aspx?f=255&MSPPError=-2147217396
I would get him to check for the dll file first and see if it is blocked.
Theoretically I could add a IO exception to the test here and simply debug.log that kind of error.
https://github.com/JimBobSquarePants/ImageProcessor/blob/d4171429ce470e890f76b0f2000120bf779791f2/src/ImageProcessor/Common/Helpers/TypeFinder.cs#L128
What do you think?

I think that would be OK. I will check the rest on my side. Thanks.

I use imageFactory.Load(imgPath); as below:
       using (ImageFactory imageFactory = new ImageFactory(preserveExifData: true))
        {
            imageFactory.Load(imgPath);
            imageFactory.ReplaceColor(source, target, 128);

            using (MemoryStream ms = new MemoryStream())
            {
                imageFactory.Save(ms);
                return Image.FromStream(ms);
            }
        }

It loads very slow on the first initialization,It takes 9965 ms to load all my assemblies which is not what I want .

ImageProcessor.Configuration.ImageProcessorBootstrapper.LoadSupportedImageFormats()  9833 ms

 System.Linq.Enumerable.ToList(IEnumerable[TSource])   7724 ms

ImageProcessor.Configuration.ImageProcessorBootstrapper+<>c.<LoadSupportedImageFormats>b__18_0(Assembly)             7718ms

 ImageProcessor.Common.Extensions.AssemblyExtensions.GetLoadableTypes(Assembly)  7718ms

 System.Reflection.Assembly.GetTypes()                              7718ms

Can you give an access to change the SupportedImageFormats individually ,so that I can set that outside without load all my assemblies or just give a flag to Load all assemblies or not . I do a little change as below:
**_public static bool LoadAssemblies=true;_**
    /// <summary>
    /// Creates a list, using reflection, of supported image formats that ImageProcessor can run.
    /// </summary>
    private void LoadSupportedImageFormats()
    {
        var formats = new List<ISupportedImageFormat>
        {
            new BitmapFormat(),
            new GifFormat(),
            new JpegFormat(),
            new PngFormat(),
            new TiffFormat()
        };

        Type type = typeof(ISupportedImageFormat);
        if (this.SupportedImageFormats == null)
        {
            _if(LoadAssemblies)
            {_
            var availableTypes =
                TypeFinder.GetAssembliesWithKnownExclusions()
                    .SelectMany(a => a.GetLoadableTypes())
                    .Where(t => type.IsAssignableFrom(t) && t.IsClass && !t.IsAbstract)
                    .ToList();

            formats.AddRange(availableTypes.Select(f => Activator.CreateInstance(f) as ISupportedImageFormat).ToList());

            this.SupportedImageFormats = formats;
            }
            else
            {
                 this.SupportedImageFormats = formats;
            }
        }
    }

`

@LavenderDaisy I'm actually looking at that code for v3.
I'd actually like to drop all automatic assembly loading and make the process entirely manual.


Add recipe for sending message to other clients.

source.id is currently broken in chrome, tracking in https://code.google.com/p/chromium/issues/detail?id=549346


[BEAM-8902] parameterize input type of Java external transform

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

R: @robertwb

Run Java PreCommit

Run Python2_PVR_Flink PreCommit

added withOutputType() which users can use to specify the element type of the output pcollection.


Map by keyPath

Hello.
Thank you for PromiseKit.
In this pull request I added new versions of map for Thenable and Guarantee. Now you can use a key path instead of a closure if you need to map a value.

Odd the code coverage dropped.


add figma resources and simplify landing page grid template

how could we have "all figma resources" link to something that has all the figma resources from our team? instead of a single designer's profile page? 😬

Related to this, we would like to post an update of Lexicon Figma Library. How do we do that?

@victorvalle what do you have in mind?

@plhnk I think that'd be ideal! The two resources on our team profile are already linked on this page so there aren't really more yet... unless you're thinking to link to the open source project. But it'd probably be better to do it all through community now we can! Any suggestions??

@abelhancock what do you think about a page at /resources/figma/designers — it would have cards for all the designers who have put their Figma community links in authors.yaml?

@plhnk I like that idea! Only downside could be lots of clicking different profiles to discover content. Would it be easy enough to create a page with cards that link directly to each resource individually? Or is that too much to maintain another collection of content?

thanks @abelhancock great update!


Add implicit conversion from T (T?) to Maybe

If T? is a struct, then it produces Maybe<Nullable> :/ use ToMaybe extension method for T? instead


chapter-make-read - Refresh the chapter list manually

Hi, I usually write down the chapter list in a text editor, as a <video>.<ext>.chp file. Is it possible to manually refresh the chapter list while playing <video>, reading from <video>.<ext>.chp?

# Mark chapters
key script-message-to chapter_make_read create_chapter
# Remove current chapter
key script-message-to chapter_make_read remove_chapter
# Edit existing chapter's title
key script-message-to chapter_make_read edit_chapter

Can these key bindings not meet editing needs? Why do you need to use a text editor?
If you only added some new chapters in the text editor, it is not difficult to implement manually refresh. But if you also delete certain chapters in the text editor, it will be difficult to implement manually refresh.

Why do you need to use a text editor?

I'm used to and I like doing it this way. I used to write down a txt with the same formatting as a chp  file, and then convert it to an EDL that just splitted the video into chapters. Then I found your script and liked it a lot, but I still also like to write down chapters in the text editor. I get to see all of them at once, I can quickly edit the titles or adjust a timestamp when I know I got it a bit too late or too soon…

If you only added some new chapters in the text editor, it is not difficult to implement manually refresh. But if you also delete some chapters in the text editor, it will be difficult to implement manually refresh.

Yes I usually don't touch the old chapters. Sometimes I change their title but I hardly ever delete them. If it's possible to update the titles of the existing chapters it would be great, but even just adding the new ones would be useful.

Yes I usually don't touch the old chapters. Sometimes I change their title but I hardly ever delete them. If it's possible to update the titles of the existing chapters it would be great, but even just adding the new ones would be useful.

The latest commit https://github.com/dyphire/mpv-scripts/commit/0881cd1390326552d113cb910ec9cc030df3aa16 has implemented this feature, please test if it works for you.

It loads new chapters and updates the title of the existing ones. It doesn't remove chapters I've deleted from the list but that's not a problem. Thanks man, great work :-)

But if you also delete some chapters in the text editor, it will be difficult to implement manually refresh.
I realize that implementing this is not difficult and have optimized the implementation in the latest commit.

Awesome!
Btw (for anyone who may stumble across this ticket), this command copies a timestamp at the current ${playback-time} in the format used by the chp file; it's what makes it easy for me to use a text editor to write down the chapter list.
Ctrl+Shift+c run "/bin/sh" "-c" "echo -n \"${playback-time/full}\" | xclip -i -selection clipboard" ; show-text "Copied timestamp for chapter-make-read.lua"

It works on GNU, I don't know about other operating systems.


added polymetis usage example

Motivation and Context

A jupyter notebook demonstrating how to use habitat with a custom controller through polymetis was created.
T122948275
How Has This Been Tested

Running the Jupyter notebook with the correct environment setup shows a waving robot in a specified scene (franka arm in an apartment with defaults)
Note that the following were added to the polymetis conda environment:
- opencv=4.5.1
- jupyter
Types of changes

New feature (non-breaking change which adds functionality)

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have completed my CLA (see CONTRIBUTING)
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

@laikhtewari Can you look into try to make this a colab notebook? See the ones defined in the examples folder for comparison.

@laikhtewari Can you look into try to make this a colab notebook? See the ones defined in the examples folder for comparison.

Agreed. We also have automation tools in place to automatically convert to a python script on commit which can be run locally and during CI testing.
Just place the colab notebook in examples/tutorials/nb_python/

Agreed. Once it's in that state, we can also add your tutorial to our current set of tutorials: https://aihabitat.org/tutorial/

Added issue to convert notebook into colab: https://github.com/facebookresearch/habitat-lab/issues/910


[FormRecognizer] Consider adding TryParse() methods to every AsX method in DocumentField

Same as https://github.com/Azure/azure-sdk-for-net/issues/15945, but for the Document Analysis Client models.

This work will be tracked as part of https://github.com/azure/azure-sdk-for-net/issues/15945 since issues are pretty much the same.


Any update on ghost v2.0

Hi
Please could you confirm if there is development to support ghost v2.0.
Thank you

FYI this one should support 2.0: https://github.com/KaySchneider/ghost-google-cloud-storage

Thank you so much.
This worked for me on App Engine Flex and GCS.
On Sun, Jun 2, 2019 at 6:07 AM Demian<EMAIL_ADDRESS>wrote:

FYI this one should support 2.0:
https://github.com/KaySchneider/ghost-google-cloud-storage
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/thombuchi/ghost-google-cloud-storage/issues/18?email_source=notifications&email_token=ABKUB7ZPV6LO67UY36GXSFTPYOLUNA5CNFSM4HRJWZ72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXSHXY#issuecomment-498017247,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKUB74PX4ZIXEF3KFQMYTDPYOLUNANCNFSM4HRJWZ7Q
.

--
Kind Regards
Abdulsalam Abdullateef
+234<PHONE_NUMBER>, +1<PHONE_NUMBER>
LinkedIn https://ng.linkedin.com/in/abdullateefabdulsalam | Twitter
https://www.twitter.com/@abcubed3


TFKUBE-182: Fix e2e test

I was waiting for result from https://github.com/atlassian-labs/data-center-terraform/actions/runs/1599664674


[Errors] wrong identifier in errors that reference indexers

var a1: { [a: string]: string };
var b1: { [b: string]: number };

var r1a1 = a1 < b1;

Expected:
!!! Operator '<' cannot be applied to types '{ [a: string]: string; }' and '{ [b: string]: number; }'.

Actual: indexer variable is always 'x'
!!! Operator '<' cannot be applied to types '{ [x: string]: string; }' and '{ [x: string]: number; }'.

Seems to be fixed
Baseline here: https://github.com/Microsoft/TypeScript/blob/master/tests/baselines/reference/comparisonOperatorWithNoRelationshipObjectsOnIndexSignature.errors.txt#L93


Inference speed issue

Hello! Thanks for your nice work. I  am tring to run FSOD evaluation demo on coco dataset. But the inference speed of the inference phase is quite low on one single 4090 GPU. The evaluation of 5000 images needs serverial hours. I have tried to decrease the number shots, but it doesn't affect much.  I find that the GPU usage is quite low (about 30%) and the memory usage is almost full. I wonder if there is a method to decrease the need of memory usage and accelerate the inference speed. Thanks again for your time.

Hi @YanxingLiu, it is possible to increase speed by reducing the K from 10 to 3 to get a large speed boost without much accuracy loss (and K=1 would be the fastest setup).
But I don't quite expect the low GPU usage, is it possible the data loading is slow because of other issues?

Owner

Thanks for your reply. I wil try to reduce the numbe of shots and figure out the reason of the low GPU usage.

Hi @YanxingLiu, it is possible to increase speed by reducing the K from 10 to 3 to get a large speed boost without much accuracy loss (and K=1 would be the fastest setup).
But I don't quite expect the low GPU usage, is it possible the data loading is slow because of other issues?

Thanks for your reply. I am trying to reduce the numbe of shots and figure out the reason of the low GPU usage. Have a nice day!

Hi @YanxingLiu,
Were you able to figure out the reason and improve the inference speed? Your answer would greatly help.
Many Thanks,


Safari - Events are displayed then moved to the right place

With Safari, when an event is clicked the details popup first appears at the top left of the screen and is then moved to the proper place in the screen.

In Safari's tech preview (10.2), this is fixed. I coudln't find a way to check with Safari 10.1 (shootout if anyone is running the latest macOS beta), but it could be fixed as well.


Added Confidence sub-product details to DEA Intertidal Extents and also the AWS link

Added the Product ID
Added a DEA Explorer link
Added an AWS link for both sub-product's data

@robbibt Would you be able to review this PR?


Error: pc-user-import.py: 400 Client Error: Bad Request for url: https://api.prismacloud.io/user

Attempting to import a number of users via a csv file and it looks like there is some client issues. Double checked my username and password (API Keys) to confirm they are correct as well as the tenant I am on.
ython pc-user-import.py ~/pc-users.csv <removed for security's sake>

Ready to execute commands against your Prisma Cloud tenant.
Would you like to continue (y or yes to continue)?y

API - Getting authentication token...Done.
API - Getting current user list...Traceback (most recent call last):
  File "pc-user-import.py", line 47, in <module>
    pc_settings, response_package = pc_lib_api.api_user_list_get(pc_settings)
  File "/Users/r630850/src/python/pc-toolbox/pc_lib_api.py", line 240, in api_user_list_get
    return pc_call_api(action, url, pc_settings)
  File "/Users/r630850/src/python/pc-toolbox/pc_lib_api.py", line 41, in pc_call_api
    response.raise_for_status()
  File "/Users/r630850/Library/Python/2.7/lib/python/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.prismacloud.io/user

The V1 API for users has been disabled on the APP stack and I was unaware.  Updated the user import to now use the V2 API for the GET list.  Re-check out and try again.  This version appears to work on all stacks now.  Also updated the CSV imported to a read only model which should help with CSV compatibility.
Let me know if this works for you.

Different error now, but I think it fixed the original error. Now getting:

Ready to execute commands against your Prisma Cloud tenant.
Would you like to continue (y or yes to continue)?y

API - Getting authentication token...Done.
API - Getting current user list...Done.
File - Loading CSV user data...Done.
API - Getting user roles...Done.
Searching for role name to get role ID...Done.
Formatting imported user list and checking for duplicates by e-mail...Traceback (most recent call last):
  File "pc-user-import.py", line 87, in <module>
    if user_new['email'].lower() == user_old['email'].lower():
KeyError: 'email'

I put a template of the CSV in the templates folder.  You need email, firstName, and lastName as the header of the CSV file so it knows what to put where.

Yep, it is setup that way, still bumping into the error.
email,firstName,lastName
<EMAIL_ADDRESS>John, Lowry

Hmmm...  That exact file works for me...  Interesting.  I have no idea why that is not working.  The only difference is Python 2 vs 3.  Do you have a Python 3 test machine anywhere?

or can you send me your test CSV (the actual file)?

Yeah, I have used both python2 and python3. Same error message.
Let me check the encoding. I can't share my csv file, it is work product and contains people's email addresses.

Can you create a copy and delete everything except line 1, and change the name/domain for like 2 entries?  I can test it on my machines as well to see if I can reproduce.  May be able to work out what is happening.

$ file -I ~/pc-users.csv
pc-users.csv: text/plain; charset=utf-8
$ file -I ~/pc-user-test.csv
pc-user-test.csv: text/plain; charset=us-ascii

I hand generated the pc-user-test.csv file with exactly one user in it. Both of them get the same error.

What OS are you using?  OSX or Linux?

OS X.

Give me an hour and I will see if I can reproduce.

For Ubuntu:
ubuntu@host:~/pc-toolbox$ python3 pc-user-import.py ./test-user-import.csv "System Admin"
Ready to execute commands against your Prisma Cloud tenant.
Would you like to continue (y or yes to continue)?y
API - Getting authentication token...Done.
API - Getting current user list...Done.
File - Loading CSV user data...Done.
API - Getting user roles...Done.
Searching for role name to get role ID...Done.
Formatting imported user list and checking for duplicates by e-mail...Done.
Users to add: 2
Users skipped (Duplicates): 1
Users removed as duplicates from CSV: 0
API - Adding users...
Adding user email:
Adding user email:
Done.
Encoding:
ubuntu@host:~/pc-toolbox$ file -i ./test-user-import.csv
./test-user-import.csv: text/plain; charset=us-ascii
Attached my test file as well.
Going to test on OSX next to see if I can find out what is happening with your import.
test-user-import.zip

For OSX:
$ python3 pc-user-import.py ./test-user-import.csv "System Admin"
Ready to execute commands against your Prisma Cloud tenant.
Would you like to continue (y or yes to continue)?y
API - Getting authentication token...Done.
API - Getting current user list...Done.
File - Loading CSV user data...Done.
API - Getting user roles...Done.
Searching for role name to get role ID...Done.
Formatting imported user list and checking for duplicates by e-mail...Done.
Users to add: 1
Users skipped (Duplicates): 1
Users removed as duplicates from CSV: 0
API - Adding users...
Adding user email:
Done.
Used the same file copied over.
$ file -I ./test-user-import.csv
./test-user-import.csv: text/plain; charset=us-ascii
Can you try downloading my test file and seeing if there is some kind of difference?  I have no idea why it would throw that error for you.

To expand on what it is doing:
The file is being loaded into a python dictionary via the csv library.  It uses the first line to determine the "keys" for the dict.  Then loads the data into the dict.  One of those keys from the first line is "email" (no caps).  Then it compares the "email" key in each record from the API pull to the "email" key from the file load to make sure there are no duplicates.  If there are duplicates, it drops them from the import before moving on.
For some reason, it seems when the file is loaded for yours, it is not loading the "email" key correctly, so when it looks for that key, it cannot find it and errors out.  But the file example you sent is structured correctly, so I have no idea why it is not importing correctly.
If the file example above fails, I can modify the code to add a debug that will dump out the imported JSON and we can see what it is actually getting for yours.

Sorry about dropping off, Dad and work duties took me away. Changed up the code to print what is going on under the hood a bit. The user it is stopping at is a pre-existing user and in my csv file. But this is what is happening:
Formatting imported user list and checking for duplicates by e-mail...
old user
<EMAIL_ADDRESS>
new user
Traceback (most recent call last):
  File "pc-user-import.py", line 92, in <module>
    print(user_new['email'])
KeyError: 'email'

snippet:
for user_old in user_list_old:
              print()
              print("old user")
              print(user_old['email'])
              print()
              print("new user")
              print(user_new['email'])
              if user_new['email'].lower() == user_old['email'].lower():
                  users_skipped_count = users_skipped_count + 1
                  user_exists = True
                  break

Wait, scratch that. I removed the old user from the import list and tweaked code a bit now I get this:
python pc-user-import.py ~/pc-users.csv prisma_healthplancombo

Ready to execute commands against your Prisma Cloud tenant.
Would you like to continue (y or yes to continue)?y

API - Getting authentication token...Done.
API - Getting current user list...Done.
File - Loading CSV user data...Done.
API - Getting user roles...Done.
Searching for role name to get role ID...Done.
Formatting imported user list and checking for duplicates by e-mail...
old user
<EMAIL_ADDRESS>
new user
{'lastName': 'Smith', 'firstName': 'John', '\xef\xbb\xbfemail'<EMAIL_ADDRESS>Traceback (most recent call last):
  File "pc-user-import.py", line 93, in <module>
    if user_new['email'].lower() == user_old['email'].lower():
KeyError: 'email'

The '\xef\xbb\xbfemail' is telling there.

That's interesting.  So it was the data it imported and a misleading error.  Once you delete that record, did it progress properly?

No, I am stilling getting:
\xef\xbb\xbfemail'

I got this list as an excel spreadsheet, exported it as a CSV and added my own headers. There has to be some weird encoding thing happening. My headers are literally "email,firstName,lastName" yet it is showing that error.

it is...:
https://stackoverflow.com/questions/18664712/split-function-add-xef-xbb-xbf-n-to-my-list
There is the header (hidden) in there that is getting imported with the first key (email).  That encoding needs to be removed.  that article gives a method.  You could also likely copy-paste the data into a "clean" shell, and that might solve it also.

Yep, just came across the same article. Removed the BOM and utility is running. Thanks for sticking with me on this.

Excellent!  Let me know if you run into any other issues.


Update to Miniquad v0.4

This PR updates the library to support Miniquad v0.4, as well as adds an example of integrating into Macroquad
Breaking changes
Event handler
Miniquad v0.4 no longer passes context as an argument for event listeners, which means that YakuiMiniQuad can no longer be an event handler directly without owning the context, which is not always possible (eg, in Macroquad), so instead I added YakuiMiniQuadOwnedHandler for when owning context is possible and YakuiMiniQuadRefHandler for when owning context is not possible, but event handler functionality is still required
Deleting textures on drop
Previously, YakuiMiniquadState deleted all managed textures on drop, but with Macroquad v0.4, all textures are managed by the context, and we only operate on IDs, so going with the drop approach is no longer possible without giving YakuiMiniquadState an owned context reference. For this reason, the drop_textures(&mut self, &mut Context) function was added to manually delete all man aged textures. YakuiMiniQuadOwnedHandler uses this function to automatically delete all managed textures on drop, since it has access to owned context.

Hi @juh9870 ! Very nice contribution, I'll have a look over it right now and see if I can update yakui-macroquad as well in the process, hopefully it's not too bad :D

Looks good to me!


Is it possible to show a fragment instead of activity for crashes?

I have a single activity based application which uses navigation components to switch between fragments. So every page is a fragment!
I need to show this error page as a fragment in my single activity.
What i need is:
1- Add crash page as fragment to my navigation graph and navigate to it.
2- Navigate to another fragment from your crash fragment with user action(Send user to support page to past the stack trace and guide the user).
Is there any possibility to do this?

It's not possible because when an app crash occurs, the process gets killed by the system. You can only launch a new Intent (that's what this library does). You could change this intent to launch a custom error activity of your own (that could be your main activity with some code to display your own error fragment), but it's impossible to keep the back stack or any state from your old activity.


Using whoami without authenticating displays an error message

If we install appix and issue the whoami command without authenticating first, it displays the following error:
2016/11/25 09:28:46 Failed to read auth file C:\Users\mvincze.DEV\.appix\auth.json
2016/11/25 09:28:46 open C:\Users\mvincze.DEV\.appix\auth.json: The system cannot find the file specified.

Instead it should display a message telling the user they're not authenticated, and they can log in with appix login.
cc @fahad19

Fixed by PR-28


Don't try to open glibc source files when debugging

Which OS: Ubuntu 18.04
Which extension version: 1.0.0
Which LLDB version: 6.0.0
Which VSCode version:  Version: 1.28.2
What is the problem and how did you get there:
I don't know which component of vscode is responsible for this, however: When debugging unit tests, there are lots of calls to assert(), and when one of these fails, the debugger tries to open the glibc source file:
Unable to open 'raise.c': File not found (file:///build/glibc-OTsEL5/glibc-2.27/sysdeps/unix/sysv/linux/raise.c).

I don't have the sources for glibc installed (I have no need for them), but vscode puts up a dialog asking me if I want to create "raise.c". Asking to create a source file for a system library is not the right answer.
You can click on the 'x' to close the dialog, but this gets pretty tedious after about the hundredth time. Worse, the dialog doesn't go away when you stop debugging, you have to click on the 'x' every single time.
It would be nice if there was a way to tell vscode never to prompt to open this file.
What did you expect to have happened instead:
What I'd like is a button telling vscode that (a) it shouldn't try and open this missing source file, and (b) don't ask for this particular source file ever again.

The error message is owned by VSCode itself, so I can't add buttons to it.
Suppression of missing source files is already implemented, VSCode should not ask you to create one unless you've changed the value of "lldb.suppressMissingSourceFiles" in configuration to "false".
If you didn't change that setting, but still see the message, I'd like to know more about your setup.

User settings:
{
    "editor.rulers": [
        100
    ],
    "editor.fontFamily": "'Hack', 'monospace'",
    "editor.fontSize": 13,
    "editor.tabSize": 2,
    "files.trimTrailingWhitespace": true,
    "terminal.integrated.fontSize": 12,
    "window.zoomLevel": 0,
    "clang.cxxflags": [
        "-I/home/talin/Projects/tempest/build/include",
        "-I/home/talin/Projects/tempest/src/compiler",
        "-I/home/talin/Projects/tempest/src/compiler_tests",
        "-I/usr/lib/llvm-6.0/include",
        "-std=c++17",
        "-Wall",
        "-Wextra",
        "-Wno-unused-parameter",
    ],
    "breadcrumbs.enabled": true
}

Workspace settings:
{
    "files.associations": {
        "typeinfo": "cpp",
        "array": "cpp",
        "bitset": "cpp",
        "initializer_list": "cpp",
        "string_view": "cpp",
        "utility": "cpp",
        "deque": "cpp",
        "list": "cpp",
        "unordered_map": "cpp",
        "unordered_set": "cpp",
        "vector": "cpp",
        "cstdio": "cpp",
        "chrono": "cpp",
        "functional": "cpp",
        "istream": "cpp",
        "ostream": "cpp",
        "ratio": "cpp",
        "type_traits": "cpp",
        "tuple": "cpp",
        "*.tcc": "cpp",
        "optional": "cpp",
        "system_error": "cpp",
        "thread": "cpp",
        "*.inc": "cpp",
        "iosfwd": "cpp",
        "*.def": "cpp"
    }
}

What's in your launch.json?

Here it is. And thanks for being so responsive :)
{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "(lldb) Compiler Unit Tests",
      "type": "cppdbg",
      "request": "launch",
      "program": "${workspaceFolder}/build/compiler_tests/compiler_tests",
      "args": ["--use-colour", "no"],
      "stopAtEntry": false,
      "cwd": "${workspaceFolder}/build",
      "environment": [],
      "externalConsole": true,
      "MIMode": "lldb"
    },
    {
      "name": "(lldb) tempestc basic.te",
      "type": "cppdbg",
      "request": "launch",
      "program": "${workspaceFolder}/build/compiler/tempestc",
      "args": ["${workspaceFolder}/build/compiler_tests/compilation/basic.te"],
      "stopAtEntry": false,
      "cwd": "${workspaceFolder}/build",
      "environment": [],
      "externalConsole": true,
      "MIMode": "lldb"
    },
    {
      "name": "(lldb) tempestc corelib",
      "type": "cppdbg",
      "request": "launch",
      "program": "${workspaceFolder}/build/compiler/tempestc",
      "args": [
        "--source-root",
        "${workspaceFolder}/src/lib",
        "${workspaceFolder}/src/lib/tempest/core"
      ],
      "stopAtEntry": false,
      "cwd": "${workspaceFolder}/build",
      "environment": [],
      "externalConsole": true,
      "MIMode": "lldb"
    },
  ]
}

"type": "cppdbg"

Ahem, I think you've filed this bug in the wrong repo.
You're welcome 🎃

That's strange. I do have your plugin installed, that's how I got the link to this github repo. I wonder how that field got populated with that value.
Oh, well, thanks anyway.


Documentation staging area and releases

Right now the release process of the documentation is the following: just merge on main and release.
It would make reasonably simpler to develop the documentation if we had a staging area, say the main branch, where we merge PRs. The main branch could be published to a "private domain", which we can access for testing that everything works fine, and then during releases we point docs.restate.dev to a specific git tag.

Just trying to understand this first. Why do we need this? If you create a PR, you see a test deployment right?
And if you merge to main, it deploys the new version (you don't need to release).

And if you merge to main, it deploys the new version (you don't need to release).

That's exactly what i would like to avoid, so when developing new features we can merge the docs, iterate on it, and not show it to the user in the meantime. This would make the release process a tad bit easier and faster, because the doc would be already (mostly) in shape when the release day comes.

Giselle and I discussed about using a new branch dev as staging area, so the workflow would be the following:

PRs will point to dev
dev is deployed continuosly on staging.docs.restate.dev or something like that
When releasing, dev is merged on main and tagged/released.
main is deployed continuosly on docs.restate.dev like it currently is

https://docs.restate.dev -> production branch
https://main--docsrestatedev.netlify.app -> main branch
We are not able to password protect only one of these without giving netlify more money. Well, technically I can probably do it with an edge function for free. But I suggest we don't bother :)

francesco and i agreed a process of tagging releases on main and pushing them to production branch to deploy a new release

We are not able to password protect only one of these without giving netlify more money. Well, technically I can probably do it with an edge function for free. But I suggest we don't bother :)

I think it's fine, some users that are on snapshots might be ok with looking at the staging docs


Do not stop watching on general overflow event

And also demote message about overflows to INFO level.

@bot-gradle test QF please.

@bot-gradle test QF please.

OK, I've already triggered QuickFeedback build for you.

OK, I've already triggered QuickFeedback build for you.


Unable to ping user with @{username} in text

With the following code:
func main() {
	client := slack.New("MY_TOKEN")

	attachment := slack.Attachment{
		Pretext:    "@robin",
		Text:       "@robin",
		MarkdownIn: []string{"@robin"},
	}

	_, _, err := client.PostMessage("#general", slack.MsgOptionAttachments(attachment))
	if err != nil {
		log.Fatal(err)
	}
}

I expect "robin" to be pinged and highlighted but it's not. Is there a way to make ping work from slack API?

Try MarkdownIn: []string{"text", "pretext"},
We've noticed that different slack clients have different default markdown behavior. So are setting MarkdownIn explicitly on all messages.

not works.
slack.go
package main

import (
	"github.com/nlopes/slack"
	"log"
	"os"
)

func main() {
	url, ok := os.LookupEnv("WEBHOOK_URL")
	if !ok {
		log.Fatal("missing WEBHOOK_URL")
	}

	attachment := slack.Attachment{
		Color:      "good",
		Text:       "@channel",
		MarkdownIn: []string{"text"},
	}
	msg := slack.WebhookMessage{
		Attachments: []slack.Attachment{attachment},
	}

	err := slack.PostWebhook(url, &msg)
	if err != nil {
		log.Fatal("failed to post slack api")
	}
}

in slack

version
$ go version
go version go1.11.5 darwin/amd64

$ cat go.mod | grep slack
	github.com/nlopes/slack v0.5.0

<!channel>
https://api.slack.com/docs/message-formatting#linking_to_channels_and_users

It works! thanks a lot

Thanks! It works.
attachment := slack.Attachment{
	Color: "good",
	Text:  "<!channel> <@jnst> test text",
}
msg := slack.WebhookMessage{
	Attachments: []slack.Attachment{attachment},
}
err := slack.PostWebhook(url, &msg)


feat: add github and workshop blocks

Closes #158
Adds 'check github' and 'check workshop' blocks.
Also corrections to button colors/fill/hover.

👍 lgtm

LGTM


dotnet-build-target-net4x-on-unix-systems

How to target net4x on Unix based systems with dotnet build
.NET Core and Mono has made it easy for us as developers to run applications on multiple platforms. But targeting multiple runtimes can sometimes still prove to be quite difficult.
https://nvd.codes/post/dotnet-build-target-net4x-on-unix-systems

test

Just testing my comment system. Will remove later.
Strike

A nice quote

another one just for testing

one
two
three

four
five
six

Testing a highlight shizzle
javascript code block


windows xp

Does flitter support Windows XP ？

See https://docs.flutter.dev/development/tools/sdk/release-notes/supported-platforms for information about supported systems.


Give user option to choose their network for setupVars file

I have multiple Wireguard networks in my house, used for home lab stuff along with various family members who need certain permissions. After installing PiVPN and using the default network (<IP_ADDRESS>) and then changing it later in the various files in /etc/pivpn etc, I kept getting network errors. However, by changing and choosing the network from the very beginning with your install script, none of those errors occurred.
I thought about making the warning more explicit and/or steering most users to using the default network, so I'm open to suggestions and changes. Very few people will actually use it, but having the flexibility is nice.
Also... this is my first PR so go easy. :) Great work on this project, though! I really love PiVPN.

There was a feature request and subsequent commits to randomize the virtual network range, so this pull request kinda overlaps, really.

We have decided to leave advanced features out of whiptail dialogs for multiple reasons, one of them is to not overwhelm less technically savvy users with information therefore causing more errors / issues during installation.

I agree with @4s3ti here.

I opened this mainly in reply to https://github.com/pivpn/pivpn/issues/1267. So given your above comment, can 1267 be closed then? Especially if you're staying away from anything advanced in whiptail?

If the problem you are trying to solve is "missing ability to specify a custom subnet", then you can already do that via unattended configuration. On the other hand if the problem is "<IP_ADDRESS>/24" overlaps with other range, this is already fixed by randomizing the range. I think it is useful to have the ability to set a custom subnet but less tech savvy probably won't do it, and more experienced uses can use the unattended setup.
Issue #1267 is about changing the subnet after the fact, not at install time. Pull request #1467 will make that easier to change the subnet by simply editing /etc/pivpn/.../setupVars.conf although it doesn't fix the issue specifically as the script still assumes /24, so the issue can't be closed.

Ah, okay, got it. I see the comment now asking about accounting for different subnet masks. Closing this and I'll open a new PR if I get it done according to #1267. Thanks for the feedback.


Target multiple components inside Popper - currently only works with single instance

I have a button component inside the popper component. Now, only the first button gets a reference but I need all the buttons in the loop to have this. How can I make all the buttons respond to popover?
vhs-button is a custom child component below.

I have multiple choices, and true\falses, that need to show the feedback with a popover and there is always more than one button. It's working otherwise.
If this doesn't make sense I'll try to make a pen for it.

Hi @chrisdel101
Please create example in jsfiddle.net
Thanks

This question was a non-starter. I tried to loop over the poppers and it did not work. Now it does. I must have had a syntax error. Being a junior dev this common. Below is what is working:

    <popper v-for="(option, i) in data" trigger="click" :options="{placement: 'top'}" ref="popOv" :disabled="disablePO" v-bind:key="i">

        {{feedback}}

      <vhs-button v-bind:key="i" slot="reference" ref="btn" :class="{'disabled': isComplete}" v-bind:text=option.displayName v-bind:icon=option.icon @click.native="check(option, $event),componentFirstClick()" @mousedown.native="disablePopover(), feedbackTrue()" @focusout.native="feedbackTrue()" />
    </popper>


fix(nimtable): fix select empty iceberg table

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
as title

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[x] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

LGTM. Could you please cherry-pick this PR to the main branch as well?

https://github.com/risingwavelabs/risingwave/pull/18449 we can merge main branch first


CASMTRIAGE-7093

Summary and Scope

Fixes: CASMTRIAGE-7093

Issue Type

Bugfix Pull Request
RFE Pull Request

The main purpose of this PR is resolving the ifcfg file generation for CASMTRIAGE-7093.
Several other updates were factored in that allow CSI to have parity with the BASH scripts in https://github.com/Cray-HPE/metal-init/pull/80
Extras:

Pretty-print data.json every time, before it was only pretty-printed after csi patch ca was invoked.
New "parent device" and "interface name" attributes for networking.IPV4Network, networking.IPV4Subnet, and initialize.NCNNetwork structs
New initialize.NCNNetwork.Gateway value
Reusable templates for DNSMasq subnets
Speed/optimization of basecamp data.json generation

Prerequisites

[ ] I have included documentation in my PR (or it is not required)
[ ] I tested this on internal system (if yes, please include results or a description of the test)
[ ] I tested this on a vshasta system (if yes, please include results or a description of the test)

Idempotency

Risks and Mitigations

This is rebased on #405, marking as a draft until #405 merges.

This was moved out of a draft today in order to ping everyone for reviews.
#405 just merged, this PR was rebased main.

I've built this branch and my generated output has STARTMODE='manual' which seems at odds with the code.
$ cat ifcfg-bond0
#
## This file was generated by cray-site-init.
#
NAME='Internal Interface'# Select the NIC(s) for access to the CRAY.

# Select the NIC(s) for access.
BONDING_SLAVE0='p1p1'
BONDING_SLAVE1='p10p1'

# Set static IP (becomes "preferred" if dhcp is enabled)
BOOTPROTO='static'
IPADDR='<IP_ADDRESS>/16'    # i.e. '<IP_ADDRESS>/20'
PREFIXLEN='16' # i.e. '20'

# CHANGE AT OWN RISK:
BONDING_MODULE_OPTS='mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer2+3'# DO NOT CHANGE THESE:

# DO NOT CHANGE THESE:
ONBOOT='yes'
STARTMODE='manual'
BONDING_MASTER='yes'


[web]: get a final working build offline

We build for mobile and standalone apk or iOS app setup file is generated. Why doesn't flutter build web give us a standalone working offline version of the website? Why do we need to have a running server to be able to run the web version of our app?

@anoureddini
in theory the could just run flutter run -d chrome --release
but if you want the "full deployment" experience
I recommend dhttpd, you can follow these

6 simple steps
##
1. [install Dart SDK](https://dart.dev/get-dart#install)
2. [update your path](https://dart.dev/tools/pub/cmd/pub-global#running-a-script-from-your-path)
3. run `pub global activate dhttpd`
4. `cd `
5. run
```console
flutter build web --dart-define=FLUTTER_WEB_USE_SKIA=true --release
cd build/web
dhttpd --port
```
6. paste in your browser `http://localhost:/`

Thanks @iapicca
Also answered in https://github.com/flutter/flutter/issues/67664

@anoureddini
in theory the could just run flutter run -d chrome --release
but if you want the "full deployment" experience
I recommend dhttpd, you can follow these
5 simple steps

Thank you @jonahwilliams and @iapicca  for your quick answers.
I think I wrote my question poorly. Let me put it this way:
This is my expectation:
After I run flutter build web I expect to be able to go to build/web and run index.html without having any server running. Since the files are static data (html & js) having a server running doesn't seem necessary.
Another version of my problem: After I build the web version, I want to zip the [project]/build/web/ folder and send it to some random non-technical guy. He must be able to run index.html and see what I have created. How is this possible?
Thanks again, in advance.

[..] After I build the web version, I want to zip the [project]/build/web/ folder and send it to some random non-technical guy. He must be able to run index.html and see what I have created. How is this possible?

@anoureddini
I'm don't think that such a feature exist for flutter or for most other frameworks for the matter,
at least I never encountered it in my very short .net career.
to show the web app externally just deploy to some service like netlify ad nuke it after a few days


Observation index table

@mimayer - Let's use this issue to discuss details about the obs index table.
I've changed the formatting a bit and also the names or required status of some of the columns:
http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/obs_index/index.html
Any objections / further suggestions so far?
Should we make it a requirement that ALT_PNT is computed at the observation mid-time, and not in some other way? I think that would be good, because people will use this for observation selection, and it would be confusing if the observation selection differs between HAP and PA because we fill mean ALT / AZ / ZEN differently.
@mimayer - Could you please take care of the TODO for ZTRGRATE?

TODO: define what “zenithed averaged mean” means or remove this column.

I've changed the formatting a bit and also the names or required status of some of the columns:

Thanks, looks much better now.

Any objections / further suggestions so far?

On first look, I saw you removed the "OBJECT" column. In my view this column should at least be optional. There are two use cases for this:

A user wants all observations on "Crab Nebula" (of course this depends on the outcome off the HESS software)
When building observation definition files I currently use the "OBJECT" column to assign a name to the observation. This is not necessarily required but very nice and convenient to have.

Should we make it a requirement that ALT_PNT is computed at the observation mid-time, and not in some other way? I think that would be good, because people will use this for observation selection, and it would be confusing if the observation selection differs between HAP and PA because we fill mean ALT / AZ / ZEN differently.

I agree that the we should use the ALT/AZ/ZEN at mid-time of the observation. Nevertheless, at least from the PA exporter, I am not sure how to do this. I will have a look but it should be trivial I hope.

Could you please take care of the TODO for ZTRGRATE?

I have looked into the PA code and could not find directly where and how this is filled. I will check again and remove it if I cannot the definition.

Update based on the feedback by @mimayer in the last comment:
https://github.com/gammapy/gamma-astro-data-formats/commit/6cbab957ae8307f05fa01c07d8bcc0f70df33b16

OBJECT as optional string column (I deleted it by accident, completely agree it's useful)
I even added recommentations on values to put ... OK?
Require that ALT/AZ/ZEN is at mid-time of the observation. This is also something we still have to fix / change in the HAP-HD exporters.

New question:
should we include info about the array location? It will be useful to
have an obs lists that can contain runs from HESS, VERITAS, CTA south / north, and a way to select out a given array, no?
Having this info is also useful for consistency checks (RA / DEC matches ALT / AZ)
Drawback is that it's an extra 3 columns (array lon, lat, height).
Maybe we can put a "OBSERVATORY" string as just one extra column and then the location can be looked up?
https://github.com/gammapy/gammapy/blob/master/gammapy/obs/observers.py

sorry I closed this by accident

OBJECT as optional string column (I deleted it by accident, completely agree it's useful) I even added recommentations on values to put ... OK?

Sounds good. Nevertheless, I would keep this as simple as possible. In PA exporters we will write to "OBJECT" whatever is stored in the HESS database. I don't think it is worth to test the object name for sanity on 'SESAME' on export run time. For example, many off observations are targeted on e.g. globular cluster with a certain name which is stored in the database.
Still, of course it is useful to put this as recommendation.

New question:
should we include info about the array location? It will be useful to
have an obs lists that can contain runs from HESS, VERITAS, CTA south / north, and a way to select out a given array, no?
Having this info is also useful for consistency checks (RA / DEC matches ALT / AZ)
Drawback is that it's an extra 3 columns (array lon, lat, height).
Maybe we can put a "OBSERVATORY" string as just one extra column and then the location can be looked up?

I am not sure about this. Since we have obs-index files per cut config, I doubt we will ever mix it for several instruments (different instruments always will have different cut configs). At the end the observatory should be handled on the level of the master index file, right?

I am not sure about this. Since we have obs-index files per cut config, I doubt we will ever mix it for several instruments (different instruments always will have different cut configs). At the end the observatory should be handled on the level of the master index file, right?

I just remembered that the EVENTS header has OBSERVER, GEOLON, GEOLAT and ALTITUDE, i.e. exactly the info I proposed to have in the obs index.
See http://gamma-astro-data-formats.readthedocs.org/en/latest/events/index.html#id2
What info to put in the index files is always a matter of convenience versus extra effort to specify / fill, i.e. to a large degree a matter of taste. Let's think about that decision a bit more ... I'm not adding it for now.
Any other comment about the obs index here welcome ... I'll leave this issue open for at least a few days ...

I just have one more comment which probably affects more the the Event specs:
http://gamma-astro-data-formats.readthedocs.org/en/latest/events/index.html
We should probably add the same optional header keywords to the event specs as for the obs-index file. This would certainly simplify the extraction of such information (as AIRPRESS, RELHUM etc.) and fill it into the obs-index file.

@mimayer - :+1: Can you adapt the events spec?
It's very convenient to be able to build the index files just from info in the FITS header, without having to inject extra info. I plan to move this script from the HD-HAP exporter to Gammapy so that it can be re-used for HAP-Fr (or even PA if you like).

@mimayer - :+1: Can you adapt the events spec?

Done.

It's very convenient to be able to build the index files just from info in the FITS header, without having to inject extra info. I plan to move this script from the HD-HAP exporter to Gammapy so that it can be re-used for HAP-Fr (or even PA if you like).

Agree this is very convenient. I am having also a script which does that. It is probably best to share this stuff somehow. Maybe this can also be independent of any high-level python package such as ctools or gammapy and could be added to the HESS software (which of course is not public). We need to discuss a strategy for this eventually - best at a phone call.

It is probably best to share this stuff somehow. Maybe this can also be independent of any high-level python package such as ctools or gammapy and could be added to the HESS software (which of course is not public). We need to discuss a strategy for this eventually - best at a phone call.

Well, we have host in HESS, and Gammapy and ctools as open-source packages, and Gammapy already has the DataStore class implemented (needs to be updated once our discussions here are settled). I don't think it makes sense to start another repo.
Your main argument for not wanting to use Gammapy was that it's hard to install, I think. But the only required dependency is Astropy and Numpy, and I don't want to write such scripts without using the Astropy Table and SkyCoord class. So from my point of view there's absolutely no point in starting another repo outside Gammapy for scripts for data handling, background modeling or whatever.

As I said, this is a separate topic...
Are there any other comments for the observation table?

There's TODOs for you to say what exactly ZTRGRATE, NSBLEVEL and RELHUM is.
Probably it's not possible / worth the effort to specify it so that it could be filled exactly equivalently by different FITS file producers in HESS (and VERITAS, ...).
So maybe instead of trying to pin this down exactly, add a comment, something like "Some HESS chains export this at the moment and this quantity can be useful for data selection. Comparing values from different chains or other telescopes would require a more specific specification."

There's a few other things above the obs spec: I have to write the time specs.
If you have time, improving the BKG_SCALE description at
http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/obs_index/index.html#notes
to a point where it's clearer how it's supposed to be used would be good.
Maybe it's enough to link to this section and say that it's supposed to be multiplied to those rates to get a better estimate?
http://gamma-astro-data-formats.readthedocs.org/en/latest/irfs/background/index.html

Done.

@lmohrmann opened #95 and we're finalising the HESS test data release which will have an observation index file, so I think we should try to conclude this discussion on observation index in this issue also.
Previously it was just @mimayer  and me, looking up I'm not sure if there's still useful bits of discussion that should be implemented.
@jknodlseder - Do you plan to keep using the observation index table format in ctools? If yes, do you have any opinions on the current format?
http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/obs_index/index.html#obs-index
@cosimonigra and @TarekHC are starting to produce index files for MAGIC - maybe you could have a look as well and comment?

My suggestion would be that we clean up http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/obs_index/index.html now without regard for backward compatibility, and then adapt our scripts / tools to a clean, from then on more stable format.
The question for time-related info is being discussed in #95 .
Otherwise - my main comment would be that I think there are way too many "required" columns. Making a valid observation index file should be as simple as possible, really I would suggest to have "OBS_ID" as only required key, i.e. to just say that at the top, but remove the sub-section distinction between required and optional columns. This is what we use internally in HESS - often time runlists are used that just have the OBS_ID as only column, so I think that should be a valid observation index table.
Alternatively, I could be persuaded that start and stop time could be declared required. Already with pointing position there might be use cases where it's not appropriate, e.g. slew runs of IACTs of other kinds of observations by telescopes that aren't fixed on a single pointing position.
Don't get me wrong - I think we should keep the definition of the pointing-related columns, so that if the info is available, it's filled in a common way instead of everyone inventing their own column names. I'm just advocating to not declare them "required" so that it becomes easier to make valid obs index tables, and the concept becomes usable for more use cases.

Hi Christoph,
I think I already said this in the past, but I personally never liked too much these tables, neither their addition to the open specs. I see them too related to the specific implementation of each science tool. Of course, I understand why they were added, as this documentation can be useful as a reference.
I would just suggest to keep in mind that the objective is to create single FITS files containing events + IRFs. In the specific case that the user wants to use another set of IRFs, then it is up to the science tools how to handle that.

@TarekHC - I disagree, there needs to be some index file in DL3 declaring what data is available.
Just look at the first CTA data that was released: 1DC
There's 1000s of runs, so any analysis has to start with run selection, and both Gammapy and ctools use index files that are roughly equivalent, but incompatible in format (and FITS as documented in this spec, the other XML that's documented in the ctools docs):

http://cta.irap.omp.eu/ctools/users/tutorials/1dc/first.html
https://nbviewer.jupyter.org/github/gammapy/gammapy-extra/blob/master/notebooks/cta_1dc_introduction.ipynb#Select-observations

It's not just up to the analyser / science tools, these index files are released with the DL3 data, and IMO there is value in having them and there would be even more value to agree on a format.
So the HESS test data release is coming, and it will have index files in this format and no XML index files so that ctools people can go analyse it straight away. I'm not even attached to the formats we have here, there's several things I dislike about them. But it seems to me that interest in collaborating on these open specs has stalled from the ctools side, so we just continue with what we have here. If the ctools XML formats were documented here and they are better in some way, we could consider deprecating / discouraging the existing formats at http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/index.html and move to simpler index files at the DL3 level. But I don't think we can get rid of them completely, i.e. CTA and other observatories just release 1000s of FITS files, letting the user scrape the FITS header infos to find the data they are interested in.
I also don't think the open spec should be limited to DL3, e.g. there's http://gamma-astro-data-formats.readthedocs.io/en/latest/skymaps/healpix/index.html which is already partly in use and continues to be extended in the Fermi science tools, and is now being implemented in gammapy.maps. This is DL4, but it would be great if ctools and others adopt and extend these formats instead of just implementing the same thing in a different format.

I would just suggest to keep in mind that the objective is to create single FITS files containing events + IRFs.

I agree with this goal. It's what we do for the upcoming HESS test data release. I strongly advocated for this for CTA 1DC, but there they put a CALDB. And apart from needing to still convince people to do this, there is the problem of duplicating IRFs and space. Especially for background models, but also for EDISP and PSF tables the responses quickly dominate over events, even if they are almost the same for many runs. So in the end I would say it's still an open question if CTA will deliver DL3 in this one file per obs, including background model, or if a way of linking to separate files will be needed.
Anyone willing to really pick up the work on the DL3 specs and organise another call to kick this off?

Yes, we have discussed this several times. :)

There's 1000s of runs, so any analysis has to start with run selection, and both Gammapy and ctools use index files that are roughly equivalent, but incompatible in format (and FITS as documented in this spec, the other XML that's documented in the ctools docs):

I understand that science tools need them, but I disagree with the fact that CTA should provide them. If you have 1000s of runs, it is extremely simple, fast and efficient to run a script (at the science tool level) to select the runs where a given source appears, or within a given time period, etc...
If science tools want to agree on a specific format for this intermediate step, of course I understand that it could be technically convenient, but not much more than that... We would still need a tool to create new data selections, so the gain is not that clear to me.

I also don't think the open spec should be limited to DL3, e.g. there's http://gamma-astro-data-formats.readthedocs.io/en/latest/skymaps/healpix/index.html which is already partly in use and continues to be extended in the Fermi science tools, and is now being implemented in gammapy.maps. This is DL4, but it would be great if ctools and others adopt and extend these formats instead of just implementing the same thing in a different format.

Yes, you have a point there. Although I see stronger (technical & scientific) reasons why we should adopt standards within higher level products as compared with adopting a standard in these "data selection" tables.

I agree with this goal. It's what we do for the upcoming HESS test data release. I strongly advocated for this for CTA 1DC, but there they put a CALDB. And apart from needing to still convince people to do this, there is the problem of duplicating IRFs and space. Especially for background models, but also for EDISP and PSF tables the responses quickly dominate over events, even if they are almost the same for many runs. So in the end I would say it's still an open question if CTA will deliver DL3 in this one file per obs, including background model, or if a way of linking to separate files will be needed.

Ok good. If you agree, then I'm totally ok with all this.
I would say that one file per obs/run is pretty clear, and will adjust the file size depending on the amount of time you want to add to each file (adding start/stop to the IRFs, you can have flexible DL3 file size). Having bulky IRFs is not optional, it is required to have small systematic uncertainties.
But with the background models I agree that it is not that clear: a simple model should be provided, but more complicated ones should be optionally generated using science tools. But I see the actual implementation of how to link each background model to each analysis not that much as a standard, but as an implementation.

Anyone willing to really pick up the work on the DL3 specs and organise another call to kick this off?

Very good question. I will see if I continue working on this topic in the middle of November. If I don't, then I really don't know who will take care of it...

If you have 1000s of runs, it is extremely simple, fast and efficient to run a script (at the science tool level) to select the runs where a given source appears, or within a given time period,

This is not true. Opening up all the FITS files and reading the headers is not lightning fast. I did it for the CTA data challenge and also for HESS:
https://github.com/gammasky/cta-dc/blob/master/data/cta_1dc_make_data_index_files.py
If the FITS files are gzipped (which I currently plan to do for the HESS test data release, see #46) it's even much more inefficient, because you have to read the complete files and decompress them just to get at the FITS headers.
If you don't provide an index file with DL3 data, then what you'll get is just lots of scripts flying around between 100s of users, with endless variants of doing glob to find files and extracting infos from FITS headers.
I agree 100% that index files are a convenience thing! It is and should be possible to analyse DL3 data without the index files! But I don't think we can or should get rid of them at the DL3 level and keep a nice user experience, I think instruments need to hand out index files of what data exists. In HESS also every analysis stars with a run selection from an observation table (in that case querying a MySQL database).
@TarekHC Maybe I just stop arguing and wait for you to do your first data release or two in MAGIC, with 100s or 1000s of runs? I'm almost certain you would also provide an observation table of what data is there, or if not most users would ask you for it. :-)

This is not true. Opening up all the FITS files and reading the headers is not lightning fast. I did it for the CTA data challenge and also for HESS:
https://github.com/gammasky/cta-dc/blob/master/data/cta_1dc_make_data_index_files.py
If the FITS files are gzipped (which I currently plan to do for the HESS test data release, see #46) it's even much more inefficient, because you have to read the complete files and decompress them just to get at the FITS headers.

Define lightning fast? It is something that you are going to do once per analysis, and it will not be that common to have that many files in the CTA era (not that many proposals with huge amount of hours).

If you don't provide an index file with DL3 data, then what you'll get is just lots of scripts flying around between 100s of users, with endless variants of doing glob to find files and extracting infos from FITS headers.

If science tools provide these scripts, then I don't see a reason why that should happen.

@TarekHC Maybe I just stop arguing and wait for you to do your first data release or two in MAGIC, with 100s or 1000s of runs? I'm almost certain you would also provide an observation table of what data is there, or if not most users would ask you for it. :-)

You might be right there... You have way more experience than I do. But even if that was the case, the debate to add those tables within the DL3 specs would still be open.
But anyway, sorry to open this debate. I do not propose to remove them from the specs, so sorry for adding noise. If they are there, we should  improve/update them! Although I'm afraid I won't be able to add much to the discussion.

If science tools provide these scripts

We had this in Gammapy, but removed it ~ 2 years ago. The problem was that the data format (folder and file and HDU names) kept changing / evolving for each new FITS production in HESS (done by 3 different people / chains), and it was annoying to have to add a new "scheme" (that's what we called it) to access the data to Gammapy for every new FITS production that came out, i.e. to touch the science tool code every time.
I would go so far as to say that archiving / releasing FITS data without index tables has been tried and was found to be very unpleasant, and this goal to have them be part of the DL3 spec (as an optional, convenience thing for data selection / access) is already based on a lesson learnt.
That said, I'm not really against re-adding parts of https://github.com/gammasky/cta-dc/blob/master/data/cta_1dc_make_data_index_files.py to Gammapy, to make it easier for people doing FITS productions for HESS, CTA, MAGIC to generate index files and checks on produced DL3 data. The problem with that is that it's hard to write this code in a re-usable way, because there's always some specific formatting that needs to be done to get to the common format. So at the moment we have the copy & paste of such index file generation scripts by the ~ 5 people in the world that do FITS productions, which is annoying but I think acceptable, as long as it's not ~ 500 CTA users that are copying around variants of that script for the FITS productions that are coming over the coming years.

But anyway, sorry to open this debate. I do not propose to remove them from the specs, so sorry for adding noise. If they are there, we should improve/update them!

I think it's OK to spend some time to re-discuss this. It's an important question and there is no great solution or concensus yet. I'm 100% agreed with this final statement: let's not remove them, but try to improve the spec a bit, since some instruments (HESS & CTA data release) and tools (Gammapy and partly ctools) support them.

it's even much more inefficient, because you have to read the complete files and decompress them just to get at the FITS headers.

Is that really true? I know libcfitsio can handle reading gzipped fits as a stream (without decompressing the full file), and I would assume the astropy.io.fits does the same.  You never need to uncompress the file manually, and the libraries only decompress the part they need (so would stop after the header).

For the question of index tables, I tend to agree with Tarek - science tools could just provide a tool to generate what is needed.  Remember in real CTA analysis, run-selection will be done mostly at the GTI level, though it is useful to have a first past to identify which files should be loaded and which IRF to use.  For that, one could have something much simpler than what is there.   Also, is there a good reason why we are not basing it on the CALDB index format (e.g. https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/data/asca/sis/)?  That would keep compatibility, and also defines how to name the IRF tables. I notice for example that this is still inconsistent between ctools and gammapy (or at least not clearly defined).

Is that really true?

I think yes. Please read and comment in #46 , which is specifically about that question.

For the question of index tables, I tend to agree with Tarek - science tools could just provide a tool to generate what is needed.

Well, then please get involved with CTA 1DC and make these comments now
https://forge.in2p3.fr/projects/data-challenge-1-dc-1/boards
and then again when the 1DC close-out document gets written / circulated.
If you don't, then 2DC and future CTA data releases will keep releasing index files in multiple formats.
:-)

I am strongly against of using a CALDB approach. In the future, we won't have like now, 1000s of runs represented by a simple IRF. Our IRFs are changing very fast, and the use of incorrect IRFs could be catastrophic. The whole DL3 approach, ideally, goes in the direction of using a single FITS file with events + IRFs, so the bond between which events are used with which IRFs is stronger (even if that link is not perfectly clear now).

In my mind, CALDB should be used as its name indicates, as a database. For Fermi-LAT the database can be included within the science tools, as the total size of all Fermi-LAT IRFs is small. For CTA, the IRF database will definitely be larger.
We could use CALDB internally to store all IRFs, but when providing the DL3 files, I would push to provide merged events+IRFs.

I am strongly against using a CALDB approach.

Me too. I was arguing we should distribute DL3 data for 1DC in a more realistic way and put run-wise IRFs. This would have shown people that IRF size matters and they need to work on good IRF formats.
I think it's important that people knowleageable about data comment on 1DC, so that there's  a chance of things getting better in the future. Probably commenting in https://forge.in2p3.fr/projects/data-challenge-1-dc-1/boards would be best. Or we could gather our comments on the 1DC dataset a bit and then post one thread listing our ~ 5 complaints with 1DC.
Roberta and I started this wiki page, where we had planned to gather such comments:
https://github.com/gammasky/cta-dc/wiki/1DC-comments-from-gammapy-team
Feel free to add things there any time if you think it's better to first gather comments and submit them all together.

I think it's important that people knowleageable about data comment on 1DC, so that there's a chance of things getting better in the future. Probably commenting in https://forge.in2p3.fr/projects/data-challenge-1-dc-1/boards would be best. Or we could gather our comments on the 1DC dataset a bit and then post one thread listing our ~ 5 complaints with 1DC.

That is actually a good idea. I have not been involved in the 1DC at all, but if I ever find the time to play with it, I will add something to the discussion.

Me too. I was arguing we should distribute DL3 data for 1DC in a more realistic way and put run-wise IRFs. This would have shown people that IRF size matters and they need to work on good IRF formats.

I can understand why these were not included for the 1DC, but in the close future we will be able to interpolate different IRFs for different azimuth/zeniths. In fact, the next data challenge could even propose the CTA MC group the challenge of generating minimal MC productions for each run following a certain path in the sky, to really encounter the problems we will find in the future.

We have no plans to support the observation index table format in ctools. As you know, I never was a friend of the index tables and I'd prefer a scheme of on-the-fly index generation which is much more flexible. Hence there should be tools that allow for example via an auto discovery to generate an index file, but the index file itself should not be part of the data format.

Le 19 oct. 2017 à 12:46, Christoph Deil<EMAIL_ADDRESS>a écrit :
@lmohrmann https://github.com/lmohrmann opened #95 https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/issues/95 and we're finalising the HESS test data release which will have an observation index file, so I think we should try to conclude this discussion on observation index in this issue also.
Previously it was just @mimayer https://github.com/mimayer and me, looking up I'm not sure if there's still useful bits of discussion that should be implemented.
@jknodlseder https://github.com/jknodlseder - Do you plan to keep using the observation index table format in ctools? If yes, do you have any opinions on the current format?
http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/obs_index/index.html#obs-index http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/obs_index/index.html#obs-index
@cosimonigra and @TarekHC https://github.com/tarekhc are starting to produce index files for MAGIC - maybe you could have a look as well and comment?
My suggestion would be that we clean up http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/obs_index/index.html http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/obs_index/index.html now without regard for backward compatibility, and then adapt our scripts / tools to a clean, from then on more stable format.
The question for time-related info is being discussed in #95 https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/issues/95 .
Otherwise - my main comment would be that I think there are way too many "required" columns. Making a valid observation index file should be as simple as possible, really I would suggest to have "OBS_ID" as only required key, i.e. to just say that at the top, but remove the sub-section distinction between required and optional columns. This is what we use internally in HESS - often time runlists are used that just have the OBS_ID as only column, so I think that should be a valid observation index table.
Alternatively, I could be persuaded that start and stop time could be declared required. Already with pointing position there might be use cases where it's not appropriate, e.g. slew runs of IACTs of other kinds of observations by telescopes that aren't fixed on a single pointing position.
Don't get me wrong - I think we should keep the definition of the pointing-related columns, so that if the info is available, it's filled in a common way instead of everyone inventing their own column names. I'm just advocating to not declare them "required" so that it becomes easier to make valid obs index tables, and the concept becomes usable for more use cases.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/issues/7#issuecomment-337871010, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2oV2Dio5hWgNrfRAwlZZUtkuVWpTJPks5styhygaJpZM4GxrWD.

I've removed the "master index table" and improved the description of the obs and HDU index table in #109 . See the new version:  http://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/index.html
I'm closing this old issue now.
There's several things that could / should be improved, but given that the index tables will likely be replaced by something new completely, I think it's OK to leave the spec for this prototype format (that has been and is in use in HESS and by other IACTs) as-is.


Creating a Job poster with no title breaks other interactions.

Description
If a Job Poster is created with no title, it breaks the other interactions within that listing within the Manager's view.
Steps to Reproduce
Steps to reproduce the behaviour. Example:

Login an a Manager
Go to https://talent.canada.ca/en/manager/jobs
Click on Preview or Edit or Edit Screening Plans
See error

Images

[ ] Regression test written.

We can probably set a default title to "Untitled Job Poster" to help alleviate this.

Is this still a thing with the job poster builder?


Unable to get ads1115 running

I want to read out a value from a ads1115 but am not able to get it compiled. The read functions always returns some error code. I assume its because of mismatching crate versions. Here is the code:
use ads1x1x::{channel, Ads1x1x, DataRate16Bit, SlaveAddr};
use embedded_hal::adc::OneShot;
use linux_embedded_hal::I2cdev;
use nb::block;

fn main()  {
    let dev = I2cdev::new("/dev/i2c-1").unwrap();
    let address = SlaveAddr::default();
    let mut adc = Ads1x1x::new_ads1115(dev, address);
    adc.set_data_rate(DataRate16Bit::Sps860).unwrap();

    let measurement = block!(adc.read(&mut channel::SingleA0)).unwrap();

    // get the I2C device back
    let dev = adc.destroy_ads1115();
}

Cargo.toml
ads1x1x = "0.2.2"
linux-embedded-hal = "0.3.2"
embedded-hal = "0.2.4"
nb = "1.0.0"

And here the error code:
error[E0034]: multiple applicable items in scope
  --> src/main.rs:86:34
   |
86 |     let measurement = block!(adc.read(&mut channel::SingleA0)).unwrap();
   |                                  ^^^^ multiple `read` found
   |
   = note: candidate #1 is defined in an impl of the trait `DynamicOneShot` for the type `Ads1x1x<DI, IC, CONV, ads1x1x::mode::OneShot>`
   = note: candidate #2 is defined in an impl of the trait `_embedded_hal_adc_OneShot` for the type `Ads1x1x<DI, IC, CONV, ads1x1x::mode::OneShot>`
help: disambiguate the method for candidate #1
   |
86 |     let measurement = block!(DynamicOneShot::read(&mut adc, &mut channel::SingleA0)).unwrap();
   |                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
help: disambiguate the method for candidate #2
   |
86 |     let measurement = block!(_embedded_hal_adc_OneShot::read(&mut adc, &mut channel::SingleA0)).unwrap();

I tried both suggested solutions but they result in some other error.
Thanks a lot for the help

Able to get it running with the newest version:
Cargo.toml
ads1x1x = "*"
nb = "*"
embedded-hal = "*"
linux-embedded-hal = "*"

extern crate ads1x1x;
extern crate embedded_hal;
extern crate linux_embedded_hal;

use ads1x1x::ic::Resolution16Bit;
use ads1x1x::mode::OneShot;
use ads1x1x::{channel, interface::I2cInterface, Ads1x1x, DataRate16Bit, SlaveAddr};
use linux_embedded_hal::{i2cdev, I2cdev};

let dev = I2cdev::new("/dev/i2c-1").unwrap();
let address = SlaveAddr::default();
let mut adc = Ads1x1x::new_ads1115(dev, address);
adc.set_data_rate(DataRate16Bit::Sps860).unwrap();
let value = block!(ads1x1x::DynamicOneShot::read(
            self.adc.as_mut().unwrap(),
            ads1x1x::ChannelSelection::SingleA0
        ))
        .unwrap();


How Print subscribe received message on web page?

I am trying to develop a web page which will print the received message from subscribing to the topic. but I am not able to do it can anyone please help me on this?

I'm trying to do a similar task by using the subscribe example. The only progress I'm getting is to set the retained flag so I'm able to receive all messages. In principle, it works BUT the loop would run forever (or at least a long time), not practical for showing up a website.
I've you're able to get the last messages and exiting right after, let me know - that's what I'm fighting for!!

Guys, could you find any way to print the recieved message? I need to parse and save data to DB, but still confused.

In principle, it works BUT the loop would run forever (or at least a long time), not practical for showing up a website.

No still no progress from my side..... I am still finding the same solution.

@TiSpace Can you please tell us some more details and the thing you did to get it?

I have the same problem，please answer，thanks

Php is not the route you want to go if you want to display the messages on a site.
Try using paho_mqtt.js for this task,  I've used it succesfully in the past to create a browser based client.
remember PHP is parsed by the server before it ever returns a line of code.

I am restarting this project so having a clear out of old issues (If you believe this is still valid please reopen)

Canned Response: Subscribe is not designed for web requests, it should be used in a long running process.
I do plan to add a subscribeAndWaitForMessage() function #95


Should mvn eclipse:eclipse build the project files?

I've been a long time Eclipse user..   Running mvn eclipse:eclipse I can import the project, but it has a lot of build errors.  These appear to be around the Scala based classes like SolrQuerySupport not being found.

I messed around for an hour with Eclipse and Maven and Scala, no joy...   Couldn't ever quite get the right incantation of versions of Scala in Eclipse and then the project to work.
I moved on to INtelliJ, which at first blush seemed much more promising.  Not an expert there, and so still having issues with having the Java side see the Scala side.
@kiranchitturi if you could share your IDE setup steps, I'd really appreciate it.  Feels like I'm close, but not quite there.   Really would love to hack on trying to pass in Solr URLS instead of ZK url on SolrCloudClient!

I haven't used eclipse in a long time, not sure of the eclipse and scala integration
For IntelliJ, it's quite easy. I blew up my project and imported again using these steps

clone the project to local machine
Make sure scala plugin is enabled for IntelliJ
In IntelliJ, click on 'File -> New -> Project from Existing Sources -> (Navigate to zeppelin-solr dir) -> select pom.xml -> click Open'
Go through the steps for creating the project
Overwrite .idea project if the IDE prompts
Once deps are resolved, click on Build Project to verify

Attaching zip of my .idea folder for reference (excluding workspace.xml)
zeppelin-solr-idea.zip
Hope this helps. It would be great to have a different option other than zkhost for building CloudSolrClient

Followed the instructions, and now I can run the unit tests!
I’ll play more with how we connect to Solr.
Thank you!

On Dec 22, 2018, at 11:34 AM, Kiran Chitturi<EMAIL_ADDRESS>wrote:
I haven't used eclipse in a long time, not sure of the eclipse and scala integration
For IntelliJ, it's quite easy. I blew up my project and imported again using these steps
clone the project to local machine
Make sure scala plugin is enabled for IntelliJ
In IntelliJ, click on 'File -> New -> Project from Existing Sources -> (Navigate to zeppelin-solr dir) -> select pom.xml -> click Open'
Go through the steps for creating the project
Overwrite .idea project if the IDE prompts
Once deps are resolved, click on Build Project to verify
Attaching zip of my .idea folder for reference (excluding workspace.xml)
zeppelin-solr-idea.zip https://github.com/lucidworks/zeppelin-solr/files/2705363/zeppelin-solr-idea.zip
Hope this helps. It would be great to have a different option other than zkhost for building CloudSolrClient
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/lucidworks/zeppelin-solr/issues/13#issuecomment-449582223, or mute the thread https://github.com/notifications/unsubscribe-auth/AABXe4NwZvd66dqrgFOGs3GTMsMyZ57cks5u7l8agaJpZM4ZfeAT.

Eric Pugh | Founder & CEO | OpenSource Connections, LLC |<PHONE_NUMBER> | http://www.opensourceconnections.com http://www.opensourceconnections.com/ | My Free/Busy http://tinyurl.com/eric-cal
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.

Okay, this ticket could be closed...   Is the configuration pretty standard for this project and any Java/Scala person would know what to do, and I'm just new to Scala and IntelliJ?   Or is there a documentation fix needed?


Please allow translations of sprites and backdrops

The sprites in the "choose sprite" menu should appear in the child's native language.
In Israel most children below the 5th grade do not know English well enough to work with English names.
Please allow translation of the backdrops and sprites names.

/cc @chrisgarrity